~95% on SimpleQA (e.g. Qwen3.6-27B on a 3090). Supports all local and cloud LLMs (llama.cpp, Ollama, Google, ...). 10+ search engines - arXiv, PubMed, your private documents. Everything Local & Encrypted.
Python academiaanthropicarxivbravedeep-researchencryptionhome-automationhomeserverlocallocal-deep-researchlocal-llmmistralollamaopenaipubmedresearchresearch-toolretrieval-augmented-generationsearxngself-hosted
"""Nothing in ``src/`` may log or serialise a whole LLM object.
Routing the Ollama bearer token through ``client_kwargs`` is the only way
it reaches the httpx clients (``ChatOllama`` has no ``headers`` field),
but ``client_kwargs`` is a plain pydantic field: langchain_ollama marks
it neither ``exclude`` nor ``repr=False``, so the token is present in
``repr(llm)``, ``llm.model_dump()`` and ``dumpd(llm)`` — the
``serialized`` payload every LangChain callback receives. There is no
supported way to hide it without subclassing a third-party model, so the
exposure is contained by contract instead.
Today the only consumer is ``metrics.token_counter.TokenCountingCallback
.on_llm_start``, which reads named scalar keys (``_type``, ``name``,
``kwargs["model"]``) and neither logs nor persists the dict.
What this contract actually detects
-----------------------------------
It is a syntactic (``ast``) rule over the shipped package, so it sees
*shapes*, not values. It reports, anywhere in a module:
1. a whole LLM object or the whole ``serialized`` dict reaching a logging
call — every ``logger.<level>(...)`` plus ``logger.bind(...)``, in a
positional argument, a keyword value, or an f-string placeholder;
2. the same values reaching a serialisation call — ``str``/``repr``/
``print``/``format``/``pprint``/``pformat``, ``dumpd``/``dumps``,
``json.dumps``/``yaml.dump``/``pickle.dumps``, and the
``.model_dump()``/``.model_dump_json()``/``.dict()``/``.json()``/
``.to_dict()`` methods on the object itself — whether or not the
result is then logged, so ``row.payload = llm.model_dump()`` counts;
3. ``<llm>.__dict__`` and the credential-bearing ``*client_kwargs``
attributes in any of those positions — including a read *into* one:
``llm.client_kwargs["headers"]``, the chained
``…["headers"]["Authorization"]`` and ``llm.client_kwargs.get(…)`` all
count, because that mapping *is* the bearer token;
4. a whole LLM object interpolated into *any* f-string, logged or not,
because the resulting string is one assignment away from a sink;
5. the ``serialized`` dict passed to any call at all, bare or wrapped in
a container (``audit([serialized])``, ``store(**{"p": serialized})``)
— the callback may read named keys off it (``serialized["_type"]``,
``serialized.get("name")``), never hand the dict on;
6. any of the above inside a comprehension, in either the produced value
or the iterable it is drawn from (``[m for m in [llm]]``).
An "LLM object" here means a bare name in ``_LLM_NAMES`` or an attribute
read whose final name is in ``_LLM_ATTRS`` (``self.llm``, ``self._llm``,
``agent.chat_model``, …) — and reading one LLM-valued attribute *off*
another (``self.llm.base_llm``, the wrapper unwrap that hands a caller
the raw model) is itself such a read.
What it does not detect
-----------------------
- **A scalar field of an LLM already identified.** ``self.base_llm.model``
and its private spellings (``_model``, ``llm_model``, ``_chat_model``)
are langchain's model-*name* string, so the chain stops there. Any other
attribute name is either credential-bearing, another handle on the
object, or outside the lists entirely.
- **Renamed or derived values.** ``x = llm`` then ``logger.info(x)``,
``payload = llm.model_dump()`` logged three functions later, a dict
literal built field-by-field from the object, or an LLM stored in a
container and read back (``llms[0]``, ``self._pool["chat"]``). There is
no dataflow analysis here. The same limit covers a credential mapping
reached through a call result (``getattr(llm, "client_kwargs")["headers"]``,
``vars(llm)["client_kwargs"]``), its ``.values()``/``.items()``/``.copy()``
views, and a bare local that holds the mapping before it is attached to
the model (``client_kwargs = {...}`` in the provider factory).
- **Third-party sinks.** Anything langchain, an SDK, a callback handler
from another package, or a tracing integration does with the object
after it leaves ``src/`` — including the ``serialized`` payload
langchain itself constructs and dispatches to every registered handler.
- **A class's own ``__str__``/``__repr__``/``__format__`` restating the
object it wraps** — ``repr``/``str``/``format`` of a whole LLM, or one
in the f-string the method returns. That does not create a sink, it
restates the exposure the contract already assumes: it does mean
``repr(RateLimitedLLMWrapper(...))`` carries the token exactly as
``repr(llm)`` does. *Only* that forwarding is exempt — a logger call, a
``model_dump``/``dumpd``/``json.dumps``/``yaml.dump``, a persistence
call, a ``__dict__`` read and a ``serialized`` hand-off are all still
reported inside such a method, as is anything in a function nested in
it. The method must be defined directly in a ``class`` body; a
module-level ``def __repr__`` is an ordinary function.
- **Code outside the shipped package**: tests, scripts and notebooks.
- **Names outside the two lists below.** The sets are literal; an LLM
held in a variable called something else is invisible to this rule.
So this is a tripwire against the idioms that would leak the token
tomorrow, not a proof that it cannot leak. The underlying exposure is
unfixed and unfixable from here without subclassing a third-party model.
``getattr(llm, "base_url", ...)`` and ``llm.some_attr`` stay allowed —
naming one attribute is not dumping the object, unless that attribute is
a credential mapping or another model — as do the identity and type
predicates ``type(llm)``, ``id(llm)``, ``isinstance(llm, X)`` and
``llm is None``, which reveal nothing.
"""
import ast
from pathlib import Path
import pytest
import local_deep_research
SRC = Path(local_deep_research.__file__).parent
# --- BEGIN PREDICATE ---
_LOG_METHODS = {
"debug",
"info",
"warning",
"warn",
"error",
"exception",
"critical",
"success",
"trace",
"log",
# loguru's contextual binder: `logger.bind(llm=llm).info(...)` puts the
# object in the record's `extra` dict, which every sink formats.
"bind",
}
# Called as plain functions: `repr(llm)`, `dumpd(llm)`. `json.dumps(x)`
# reaches this rule through _DUMP_METHODS instead.
_DUMP_FUNCS = {
"str",
"repr",
"format",
"print",
"pprint",
"pformat",
"dumpd",
"dumps",
}
# Called as methods. Two distinct positions matter: the *receiver*
# (`llm.model_dump()`) and the *arguments* (`json.dumps(serialized)`,
# `yaml.dump(llm)`).
_DUMP_METHODS = {
"dump",
"dumps",
"dumpd",
"model_dump",
"model_dump_json",
"dict",
"json",
"to_dict",
"to_json",
"serialize",
}
# Bare variable names that hold a whole LLM.
_LLM_NAMES = {
"llm",
"_llm",
"base_llm",
"raw_llm",
"chat_model",
"chat_llm",
"wrapped_llm",
"model_instance",
"llm_instance",
"provider_llm",
}
# Attribute names that hold langchain's model-*name* string rather than a
# model object: `ChatOllama.model`, `ChatOpenAI.model`, and the private
# spellings this tree uses for the same value. Read off an LLM this rule
# has already identified (`self.base_llm.model`), one of these is a single
# scalar field and the chain stops there; read off anything else
# (`strategy.model`) the owner is unknown and the value is taken to be the
# object.
_SCALAR_LLM_FIELDS = {
"model",
"_model",
"llm_model",
"_chat_model",
}
# Attribute names that hold a whole LLM: `self.llm`, `handler._llm`,
# `strategy.model`. Checked on the *final* name of the attribute chain,
# so the owner can be anything.
_LLM_ATTRS = _LLM_NAMES | _SCALAR_LLM_FIELDS
# The callback payload. Named keys may be read off it; the dict itself
# may not be logged, serialised, or handed to another call.
_SERIALIZED = "serialized"
_CREDENTIAL_ATTRS = {
"client_kwargs",
"sync_client_kwargs",
"async_client_kwargs",
}
# Mapping reads that are equivalent to a subscript: `d.get("k")` and
# `d["k"]` return the same value, so both are followed to their base.
_MAPPING_READS = {"get", "setdefault", "pop"}
# Reveal nothing about the object's contents, so an LLM reaching one of
# these is not a leak — and neither is whatever the call returns.
_SAFE_FUNCS = {
"type",
"id",
"isinstance",
"issubclass",
"callable",
"hasattr",
"len",
"bool",
"getattr",
}
def _llm_label(node):
"""Return a printable name if ``node`` is a whole LLM / the payload."""
if isinstance(node, ast.Name):
if node.id in _LLM_NAMES:
return node.id
if node.id == _SERIALIZED:
return _SERIALIZED
return None
if isinstance(node, ast.Attribute) and node.attr in _LLM_ATTRS:
# `self.base_llm.model` is langchain's model-*name* string, a
# scalar field of an LLM we have already identified — not a
# second LLM — so that one chain stops here. Every other name in
# the set holds an object, and reading it off an LLM is exactly
# the wrapper unwrap (`self.llm.base_llm`) that hands a caller
# the credential-bearing model: those keep the label.
if (
node.attr in _SCALAR_LLM_FIELDS
and _llm_label(node.value) is not None
):
return None
return f".{node.attr}"
return None
def _is_safe_call(node):
"""``type(llm)``/``isinstance(llm, X)``/``getattr(llm, "base_url")``.
``getattr`` is exempt only when the attribute it names is not itself
credential-bearing.
"""
if not (isinstance(node, ast.Call) and isinstance(node.func, ast.Name)):
return False
if node.func.id not in _SAFE_FUNCS:
return False
if node.func.id == "getattr" and len(node.args) >= 2:
wanted = node.args[1]
if (
isinstance(wanted, ast.Constant)
and wanted.value in _CREDENTIAL_ATTRS
):
return False
return True
def _is_identity_compare(node):
"""``llm is None`` / ``llm in registry`` — a boolean, not a dump."""
return isinstance(node, ast.Compare) and all(
isinstance(op, (ast.Is, ast.IsNot, ast.In, ast.NotIn))
for op in node.ops
)
def _indexed_base(node):
"""Follow a ``[...]``/``.get(...)`` chain down to what is indexed.
``llm.client_kwargs["headers"]`` *is* ``{"Authorization": "Bearer …"}``
— indexing a credential mapping is not reading one named scalar off
the object, so the chain is followed to its base and reported.
``serialized["_type"]`` and ``serialized.get("name")`` bottom out in
the callback payload, which the contract explicitly allows to be read
by key, so that base returns nothing.
"""
depth = 0
while True:
if isinstance(node, ast.Subscript):
node, depth = node.value, depth + 1
continue
if (
isinstance(node, ast.Call)
and isinstance(node.func, ast.Attribute)
and node.func.attr in _MAPPING_READS
):
node, depth = node.func.value, depth + 1
continue
break
if depth == 0:
return None
if isinstance(node, ast.Attribute) and node.attr in _CREDENTIAL_ATTRS:
owner = _llm_label(node.value)
return None if owner is None else f"{owner}.{node.attr}"
label = _llm_label(node)
return None if label in (None, _SERIALIZED) else label
def _payload_leaks(node):
"""Whole-object leaks reachable from a value that *is* being emitted.
Descent stops at a named scalar attribute read (``llm.model_name`` is
one string, not the object) and at a subscript of anything but a
credential mapping or an LLM (``serialized["_type"]``).
"""
if node is None or _is_safe_call(node) or _is_identity_compare(node):
return []
label = _llm_label(node)
if label is not None:
return [(node.lineno, f"`{label}`")]
if isinstance(node, ast.Attribute):
owner = _llm_label(node.value)
if owner is not None and (
node.attr in _CREDENTIAL_ATTRS or node.attr == "__dict__"
):
return [(node.lineno, f"`{owner}.{node.attr}`")]
# A named attribute of an LLM is allowed; keep looking inside
# anything that is not itself a plain name/attribute chain.
if isinstance(node.value, (ast.Name, ast.Attribute)):
return []
return _payload_leaks(node.value)
if isinstance(node, ast.Subscript):
base = _indexed_base(node)
if base is not None:
return [(node.lineno, f"`{base}` is read by key")]
return []
if isinstance(node, ast.IfExp):
# `'yes' if llm else 'no'` — the test is a truthiness guard in the
# same class as `llm is None`; only the branches are emitted.
return _payload_leaks(node.body) + _payload_leaks(node.orelse)
if isinstance(node, ast.UnaryOp) and isinstance(node.op, ast.Not):
return []
if isinstance(node, ast.Call):
base = _indexed_base(node)
if base is not None:
return [(node.lineno, f"`{base}` is read by key")]
out = []
for arg in node.args:
out += _payload_leaks(arg)
for kw in node.keywords:
out += _payload_leaks(kw.value)
out += _payload_leaks(node.func)
return out
if isinstance(node, ast.comprehension):
# The loop target is a fresh binding, not a read of the object;
# the iterable and the guards are ordinary reads.
out = _payload_leaks(node.iter)
for cond in node.ifs:
out += _payload_leaks(cond)
return out
# No `ast.expr` filter: `ast.comprehension` is not an expression, and
# skipping it left `[m for m in [llm]]` unvisited below its `elt`.
out = []
for child in ast.iter_child_nodes(node):
out += _payload_leaks(child)
return out
def _serialized_reads(node):
"""Bare ``serialized`` reads in a value handed to a call.
Wrapping it changes nothing: ``audit([serialized])``,
``audit({"p": serialized})`` and ``store(**{"p": serialized})`` hand
the dict on exactly as ``audit(serialized)`` does. A read *by key*
inside the container (``{serialized["_type"]: 1}``) is still allowed.
"""
if isinstance(node, ast.Name):
return [node] if node.id == _SERIALIZED else []
if isinstance(node, (ast.List, ast.Tuple, ast.Set, ast.Dict)):
out = []
for child in ast.iter_child_nodes(node):
out += _serialized_reads(child)
return out
return []
def _call_leaks(node):
"""Node-local rules for a single ``ast.Call``."""
func = node.func
out = []
args = list(node.args) + [kw.value for kw in node.keywords]
is_log = isinstance(func, ast.Attribute) and func.attr in _LOG_METHODS
if is_log:
for arg in args:
out += [
(line, f"{what} reaches `.{func.attr}(...)`")
for line, what in _payload_leaks(arg)
]
is_dump_func = isinstance(func, ast.Name) and func.id in _DUMP_FUNCS
is_dump_method = (
isinstance(func, ast.Attribute) and func.attr in _DUMP_METHODS
)
if is_dump_func or is_dump_method:
sink = func.id if is_dump_func else f".{func.attr}"
for arg in args:
out += [
(line, f"{what} is serialised by `{sink}(...)`")
for line, what in _payload_leaks(arg)
]
if is_dump_method:
# The receiver: `llm.model_dump()`, `self.llm.dict()`.
receiver = _llm_label(func.value)
if receiver is not None:
out.append(
(
node.lineno,
f"`{receiver}` is serialised by `.{func.attr}()`",
)
)
# The callback payload may be read by key, never handed on.
if not (is_log or _is_safe_call(node)):
for arg in args:
for name in _serialized_reads(arg):
out.append(
(
name.lineno,
f"`{_SERIALIZED}` is passed whole to another call",
)
)
return out
# A wrapper's own ``__repr__`` forwarding to the object it wraps does not
# create a sink: it makes ``repr(wrapper)`` exactly as revealing as
# ``repr(llm)``, which is the exposure this contract already assumes and
# guards. ``RateLimitedLLMWrapper.__str__``/``__repr__``
# (``web_search_engines/rate_limiting/llm/wrapper.py``) is the site this
# carve-out exists for, but these names are common — 62 of them across 22
# files, mostly SQLAlchemy models — so the exemption is kept as narrow as
# the justification: it covers only the forwarding stringify itself
# (``repr``/``str``/``format`` of an LLM, or an LLM in the f-string the
# method returns), only in a method defined directly in a ``ClassDef``
# body, and not in any function nested inside it. A logger call, a
# ``model_dump``, a persistence call or a ``serialized`` hand-off is a
# real sink wherever it is written.
_REPR_DUNDERS = {"__str__", "__repr__", "__format__"}
# Restating the wrapped object's own text is what these methods are for.
_FORWARDING_REPR_FUNCS = {"str", "repr", "format"}
def _own_scope_nodes(func):
"""Every node of ``func`` that is not inside a nested scope."""
inside = set()
pending = list(ast.iter_child_nodes(func))
while pending:
node = pending.pop()
inside.add(id(node))
if isinstance(
node,
(
ast.FunctionDef,
ast.AsyncFunctionDef,
ast.Lambda,
ast.ClassDef,
),
):
# A nested `def helper(): print(llm)` is ordinary code that
# happens to sit in a dunder; it gets no carve-out.
continue
pending += list(ast.iter_child_nodes(node))
return inside
def _repr_dunder_nodes(tree):
"""Nodes in a class's own ``__str__``/``__repr__``/``__format__``.
The method must be a direct child of a ``ClassDef`` body: a
module-level ``def __repr__`` is an ordinary function, and naming one
of these must not switch the contract off for a whole module.
"""
inside = set()
for owner in ast.walk(tree):
if not isinstance(owner, ast.ClassDef):
continue
for node in owner.body:
if (
isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef))
and node.name in _REPR_DUNDERS
):
inside |= _own_scope_nodes(node)
return inside
def _is_forwarding_repr(node):
"""``repr(self.base_llm)`` — the wrapped object restating itself.
Only a whole LLM qualifies: ``repr(llm.client_kwargs)`` is a
credential read wearing a repr, and ``str(serialized)`` is the
callback payload. Neither is what a wrapper's dunder is for.
"""
if not (
isinstance(node.func, ast.Name)
and node.func.id in _FORWARDING_REPR_FUNCS
):
return False
return bool(node.args) and _is_forwarded_value(node.args[0])
def _is_forwarded_value(node):
"""A whole LLM, or its repr — all a wrapper's dunder may restate."""
if isinstance(node, ast.Call):
return _is_forwarding_repr(node)
label = _llm_label(node)
return label is not None and label != _SERIALIZED
def _violations(source: str, label: str):
"""Yield ``"<label>:<line> <reason>"`` for every offending sink."""
tree = ast.parse(source, filename=label)
exempt = _repr_dunder_nodes(tree)
found = []
for node in ast.walk(tree):
in_repr = id(node) in exempt
if isinstance(node, ast.Call):
# Inside a repr dunder the wrapped object's own text may be
# restated — and nothing else. Every other sink stays live.
if in_repr and _is_forwarding_repr(node):
continue
found += _call_leaks(node)
elif isinstance(node, ast.JoinedStr):
# An f-string is one assignment away from any sink, so the
# interpolation itself is the violation — except where such a
# method restates the wrapped object, which is the f-string
# `f"RateLimitedLLMWrapper({repr(self.base_llm)})"` returns.
for part in node.values:
if not isinstance(part, ast.FormattedValue):
continue
if in_repr and _is_forwarded_value(part.value):
continue
found += [
(line, f"{what} is interpolated into an f-string")
for line, what in _payload_leaks(part.value)
]
elif isinstance(node, ast.Attribute) and node.attr == "__dict__":
owner = _llm_label(node.value)
if owner is not None:
found.append((node.lineno, f"`{owner}.__dict__` is read"))
seen = set()
out = []
for line, reason in found:
key = (line, reason)
if key in seen:
continue
seen.add(key)
out.append(f"{label}:{line} {reason}")
return sorted(out)
# --- END PREDICATE ---
class TestTheGuardActuallyDetects:
"""A contract test that cannot fail is worthless; prove it fires."""
@pytest.mark.parametrize(
"snippet",
[
'logger.info(f"built {llm}")',
"logger.debug(llm)",
'logger.info("cfg %s", llm.client_kwargs)',
"json.dumps(serialized)",
"logger.warning(repr(llm))",
'logger.info(f"start {serialized}")',
# Shapes the first round of this contract missed.
'logger.info(f"{repr(llm)}")',
'logger.info(f"{llm.model_dump()}")',
'logger.info("%s", llm.model_dump())',
"row.payload = llm.model_dump()",
"store(dumpd(llm))",
"session.add(Row(payload=serialized))",
'logger.info(f"{self.llm}")',
'logger.bind(llm=llm).info("made")',
'msg = f"llm={llm}"',
"print(llm)",
# Neighbours of the above.
'logger.info(f"{self._llm}")',
'logger.debug(f"{self.model}")',
"yaml.dump(llm)",
"pickle.dumps(self.chat_model)",
"payload = self.llm.__dict__",
'logger.info("cfg %s", self.llm.sync_client_kwargs)',
'logger.info(getattr(llm, "client_kwargs", None))',
"audit(serialized)",
"logger.bind(payload=serialized).debug('x')",
"return llm.model_dump_json()",
"blob = self.base_llm.to_dict()",
# Indexing a credential mapping reads the token itself.
'logger.error("cfg %s", llm.client_kwargs["headers"])',
"logger.error(f\"{llm.client_kwargs['headers']}\")",
'logger.info(llm.client_kwargs["headers"]["Authorization"])',
"logger.info(f\"{llm.sync_client_kwargs['headers']}\")",
"logger.info(f\"{llm.client_kwargs.get('headers')}\")",
'logger.info(f"{llm.async_client_kwargs[0]}")',
# Unwrapping a wrapper hands over the wrapped model.
'logger.info(f"{self.llm.base_llm}")',
'logger.info(f"{llm.base_llm}")',
'logger.info(f"{self.model.base_llm}")',
'logger.info(f"{self.llm.base_llm.client_kwargs}")',
"self.llm.base_llm.model_dump()",
"repr(self.llm.base_llm)",
# A comprehension's iterable is read like any other value.
'logger.info(f"{[m for m in [llm]]}")',
'logger.info(", ".join(str(m) for m in [llm]))',
# Wrapping the payload does not stop it being handed on.
"audit([serialized])",
'audit({"p": serialized})',
'store(**{"p": serialized})',
# The repr-dunder exemption covers the forwarded repr and
# nothing else: not a logger, not a dump, not persistence,
# not the payload, not a nested `def`, not a bare function.
"class W:\n"
" def __repr__(self):\n"
" logger.info(self.base_llm)\n"
" return 'W'",
"class W:\n"
" def __repr__(self):\n"
" session.add(Row(p=self.llm.model_dump()))\n"
" return 'x'",
"class W:\n"
" def __repr__(self):\n"
" audit(serialized)\n"
" return 'x'",
"class W:\n"
" def __format__(self, spec):\n"
" store(dumpd(llm))\n"
" return 'x'",
"class W:\n"
" def __repr__(self):\n"
" def helper():\n"
" return repr(llm)\n"
" return 'x'",
"class NotAnLLM:\n"
" def __repr__(self):\n"
" yaml.dump(llm)\n"
" return 'x'",
"def __repr__(self):\n return repr(self.llm)",
"class W:\n"
" def __repr__(self):\n"
" return repr(llm.client_kwargs)",
],
ids=(
"fstring-llm",
"bare-llm",
"client-kwargs-attr",
"json-dumps-serialized",
"repr-llm",
"fstring-serialized",
"nested-repr-in-fstring",
"model-dump-in-fstring",
"model-dump-printf-arg",
"model-dump-persisted",
"dumpd-into-a-store",
"serialized-persisted",
"self-llm-in-fstring",
"logger-bind-kwarg",
"fstring-outside-a-call",
"print-llm",
"self-underscore-llm",
"self-model-attr",
"yaml-dump",
"pickle-dumps-attr",
"dunder-dict",
"credential-attr-on-self",
"getattr-credential-attr",
"serialized-to-any-call",
"bind-serialized",
"model-dump-json-returned",
"to-dict-on-attr",
"credential-subscript",
"credential-subscript-in-an-fstring",
"credential-subscript-chained",
"sync-credential-subscript",
"credential-get",
"async-credential-index",
"wrapper-unwrap-off-self-llm",
"wrapper-unwrap-off-a-bare-llm",
"wrapper-unwrap-off-self-model",
"credential-behind-an-unwrap",
"model-dump-behind-an-unwrap",
"repr-of-an-unwrap",
"comprehension-iterable",
"generator-iterable",
"serialized-in-a-list",
"serialized-in-a-dict",
"serialized-in-a-splat",
"logger-inside-a-repr-dunder",
"persistence-inside-a-repr-dunder",
"serialized-inside-a-repr-dunder",
"dumpd-inside-a-format-dunder",
"nested-def-inside-a-repr-dunder",
"repr-dunder-of-an-unrelated-class",
"module-level-repr-dunder",
"credential-repr-inside-a-repr-dunder",
),
)
def test_offending_shapes_are_reported(self, snippet):
assert _violations(snippet, "<snippet>")
@pytest.mark.parametrize(
"snippet",
[
"logger.info(f\"url {getattr(llm, 'base_url', None)}\")",
'logger.debug(f"model {llm.model_name}")',
'model_name = serialized["_type"]',
'logger.info("count %d", len(prompts))',
# Identity and type predicates reveal nothing (R2.4).
'logger.info(f"{type(llm).__name__}")',
'logger.info(f"{id(llm)}")',
'logger.info(f"{llm is None}")',
'logger.info("%s", isinstance(llm, ChatOpenAI))',
"logger.debug(f\"{serialized.get('name')}\")",
'logger.info("%s", serialized["kwargs"]["model"])',
'logger.debug(f"{llm.__class__.__name__}")',
"if llm is not None:\n logger.info('ready')",
'json.dumps({"model": llm.model_name})',
# Truthiness guards and a wrapper restating its own repr.
"logger.debug(f\"llm={'yes' if self.llm else 'no'}\")",
'logger.debug(f"{not self.llm}")',
"model = str(self.base_llm.model)",
'logger.info(f"{self.llm.model}")',
'logger.debug(f"{self.base_llm._model}")',
"class W:\n"
" def __repr__(self):\n"
" return f'W({repr(self.base_llm)})'",
'class W:\n def __repr__(self):\n return f"W({self.base_llm!r})"',
"class W:\n def __str__(self):\n return str(self.base_llm)",
# A comprehension over something that is not an LLM, and the
# payload still read by key inside a container.
'logger.info(f"{[p for p in prompts]}")',
'logger.info(f"{[m for m in pool if m is llm]}")',
'audit({serialized["_type"]: 1})',
],
ids=(
"getattr-one-attribute",
"one-named-attribute",
"subscript-outside-a-log-call",
"unrelated-log",
"type-name",
"id-of",
"is-none",
"isinstance",
"serialized-get-named-key",
"serialized-nested-key",
"class-name",
"identity-guard",
"one-attribute-into-json",
"truthiness-guard-in-a-conditional",
"negated-truthiness-guard",
"model-name-field-of-an-llm",
"model-field-of-an-identified-llm",
"private-model-field-behind-an-unwrap",
"wrapper-repr-dunder",
"wrapper-repr-dunder-conversion",
"wrapper-str-dunder",
"comprehension-without-an-llm",
"identity-guard-in-a-comprehension",
"serialized-key-inside-a-container",
),
)
def test_legitimate_shapes_are_not_reported(self, snippet):
assert not _violations(snippet, "<snippet>")
class TestNoSinkInSource:
def test_no_module_logs_or_serialises_a_whole_llm(self):
"""Grep-style contract over the shipped package."""
# Given
modules = sorted(SRC.rglob("*.py"))
assert len(modules) > 100, "source tree not found"
# When
offenders = []
for path in modules:
offenders += _violations(
path.read_text(encoding="utf-8"),
str(path.relative_to(SRC)),
)
# Then
assert offenders == [], (
"An LLM object or a callback `serialized` payload reaches a log "
"or serialisation sink. The Ollama bearer token lives in "
"`client_kwargs`, so that sink would persist a credential:\n"
+ "\n".join(offenders)
)