Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.
Python cliconsumer-gpudpofine-tuningggufhuggingfacellmllmopslocal-ailocal-llmloralow-vrammachine-learningollamapeftpythonpytorchqlorasfttransformers
"""Issue #748 — a config field that reaches no consumer must fail the suite.
A field is declared in `config/schema.py`, validated, documented with a worked
example, and read by nothing. It accepts a value and does nothing with it,
silently. Searching this tracker for `ignore|silently|never read|does not
honour|no caller` returns 36 issues, among them #683, #684, #685 and #686 --
four in a single backend. Every one was found by a person reading code.
Nothing failed when a field lost its last consumer, and nothing fails today
when a field is added with no wiring. This guard closes that.
**What it does and does not claim.** It answers "does anything read this",
not "does this backend read this". `training.max_grad_norm` is read by sixteen
transformers trainers and by nothing on MLX; that is a strictly harder problem
and out of scope here.
**What this cannot see, so nobody trusts it past its limits.**
*The name space is global.* It asks whether an identifier appears as an
attribute or key anywhere under ``src/``, not whether it appears on a config
object. The consumed set is roughly 3,400 names -- most of the codebase's
attribute namespace. A field named after a common attribute therefore reads as
consumed on the strength of an unrelated one. Measured on this tree, an
unwired field would be caught or missed as follows:
training.verbose caught
training.top_k LEAKS -- `top_k` is an attribute elsewhere
training.top_p LEAKS
training.temperature LEAKS -- `request.temperature`, commands/serve.py
training.dtype LEAKS
training.seed LEAKS
training.logging_steps LEAKS
So this is a ratchet with a known hole, not a proof. It leaks hardest on
exactly the generic names a new HuggingFace/TRL passthrough field would carry,
which is the case most likely to arise. `test_the_known_leak_is_still_the_
known_leak` pins the boundary so it cannot move without someone noticing.
Scoping reads to config-typed objects is a much larger piece of work and is
not attempted here.
*Fields read only through a ``schema.py`` ``@property`` ARE now seen*, but
only when something calls the property. ``schema.py`` is excluded from the
main scan, so ``training.bnb_4bit_use_double_quant`` -- resolved by the
``double_quant_on`` property (``schema.py:1880``, #321) -- read as an orphan,
and an earlier version of this allowlist recorded it as an unread offender on
exactly that evidence. Freezing a repaired field as an open defect is the
worst thing an allowlist can do, because no test can retire it: the list is
meant to name fields a user can set with no effect, and an entry that
contradicts itself gives a false answer to the one question the file exists to
answer. `schema_property_reads` fixes it. The gate matters as much as the
pass: an UNCALLED resolver launders nothing, or the detector's own failure
mode returns one level up, with a field "read" by code that never runs.
Validators are excluded on principle -- a validator checks a value, a property
resolves one for a consumer.
*A read inside a function nothing references is not consumption* (#807), and
the gate that enforces it has limits in both directions. A field is dropped
only when EVERY read sits in such a function; one live read anywhere rescues
it, which is what keeps the impact small -- on this tree it drops exactly one
declared field, `warmup_auto`, pinned by
`test_the_real_tree_has_exactly_one_known_escape`.
"Referenced" is a syntactic question, not a reachability one, and it errs
toward *referenced* -- so distrust a pass more than a failure. Two leaks
follow. A function called only from ANOTHER dead function counts as
referenced, because the dead caller's body loads its name: deadness is not
transitive here. And the match is global by spelling, so an unrelated
`obj.name` anywhere under ``src/`` rescues a dead function of the same name.
The opposite hole is the one to watch: a function whose name is never loaded
is not referenced at all, so its reads are dropped and a live field could be
reported unconsumed. Typer command bodies are its largest decorated class
among the modules the gate scans; undecorated functions are a larger share.
`@app.command()` loads `app` and `command`, never the decorated function's own
name, so on this tree 115 of the 150 `*.command`-decorated functions are absent
from `referenced_names`, and the other 35 are present only through the global
spelling match above. A function reached only through a string --
`getattr(module, "name")`, or an entry point declared outside ``src/`` -- is the
same case. On this tree that drops nothing beyond the four above.
**Two rules this file learned the hard way, kept because they generalise.**
A check can only be pinned by testing its REFUSALS: loosening a predicate
makes a suite pass rather than fail, so `_reason_is_accountable` is asserted
against what it rejects. And an assertion that looks untestable usually is not
-- it is only unextracted. The scan/import tree check in `_declared()` was
described here as "unkillable by construction", on the grounds that it exists
to fire in a misconfigured environment and no mutation run inside a correct
one can reach it. That was true of the assertion as written and false of the
question it asks: pulling the comparison out as `describe_tree_mismatch` made
it an ordinary function over two paths, testable in both directions with
`tmp_path` and killed by three mutations. The lesson is the more useful one:
"untestable from here" is a claim about the current shape of the code, not
about the property.
*It cannot see a value that is read and then rewritten.* #423 is the shape:
``detect_device()`` did not recognise MLX, so `quantization: 4bit` was read
correctly and then silently rewritten to `none`. That needs device-aware
expectations, not a reachability walk.
**Why an AST walk and not a grep.** `citation_recall_threshold` appears in a
validator's error-message strings, so `grep -rl` calls it consumed while
nothing applies it. Docstrings are stripped before the walk for the same
reason. Conversely a field reached only through `getattr(cfg, "name")` or
`cfg_dict["name"]` IS consumed, and a guard that flagged those would be
deleted within a week -- `relora_steps` and `loraplus_lr_ratio` are exactly
that shape. Both directions are pinned below.
"""
from __future__ import annotations
import ast
import functools
import os
import pathlib
import re
import types
import pytest
from tests._source_scan import memoized_scan, parse_source, walk_nodes
pytestmark = pytest.mark.unit
SRC = pathlib.Path(__file__).resolve().parents[1] / "src" / "soup_cli"
SCHEMA = "schema.py"
SCHEMA_PATH = SRC / "config" / SCHEMA
# --------------------------------------------------------------------------
# The detector. Kept here rather than in `src/` because it is test-only
# tooling; nothing in the shipped CLI should depend on it.
# --------------------------------------------------------------------------
@functools.lru_cache(maxsize=None)
def _consumed_cached(paths_key) -> frozenset:
return frozenset(consumed_names([pathlib.Path(p) for p in paths_key]))
def consumed_names(paths) -> set:
"""Names some module READS. Reads only -- writes are not consumption.
Counted as a read:
* `obj.field` in a load context;
* `d["field"]` in a load context;
* `getattr(obj, "field")` / `cfg.get("field")` and friends.
Deliberately NOT counted:
* `d["field"] = value` and `{"field": value}` -- that is code EMITTING
config, not reading the user's setting. This is the hole that let
both of the maintainer's named offenders through: `data.interleave`
looked consumed because `mix_proxy.py` writes
`data_block["interleave"] = {...}`, and
`bnb_4bit_use_double_quant` because `save_formats.py` writes it as a
key in an output dict. Run against the tree at the commit where each
was a live defect, the earlier version reported both as CONSUMED.
* docstring prose, and any other bare string constant. Nothing here
collects a free-standing `ast.Constant`, so prose is excluded
structurally rather than by a stripping pass. An earlier version
stripped docstrings explicitly; mutation testing showed that pass was
dead once reads were narrowed to Load contexts and call arguments, so
it was removed rather than left looking load-bearing.
"""
names: set = set()
for path in paths:
try:
tree = parse_source(path.read_text(errors="ignore"), keep=True)
except (SyntaxError, UnicodeDecodeError, ValueError):
continue
for node in walk_nodes(tree):
if isinstance(node, ast.Attribute):
if isinstance(node.ctx, ast.Load):
names.add(node.attr)
elif isinstance(node, ast.Subscript):
sl = node.slice
if (
isinstance(node.ctx, ast.Load)
and isinstance(sl, ast.Constant)
and isinstance(sl.value, str)
):
names.add(sl.value)
elif isinstance(node, ast.Call):
# getattr(obj, "field") / d.get("field") / pop / setdefault
fn = node.func
fname = fn.attr if isinstance(fn, ast.Attribute) else getattr(fn, "id", "")
# Only the NAME argument, never the default. `d.get("k", "widget")`
# returns "widget" as a fallback value; counting it as a read of a
# field called `widget` is a false positive, and false positives
# are what get a guard deleted.
idx = 1 if fname in ("getattr", "hasattr") else 0
if fname in ("getattr", "get", "pop", "setdefault", "hasattr"):
if len(node.args) > idx:
arg = node.args[idx]
if isinstance(arg, ast.Constant) and isinstance(arg.value, str):
names.add(arg.value)
return names
def training_receiver_reads(paths, field: str) -> bool:
"""Whether ``field`` is read directly from a ``*.training`` receiver.
The global detector sees ``ship.py``'s unrelated local named
``forgetting_threshold``. This narrower check proves that exception instead
of permanently suppressing the config field: the moment a consumer reads
``cfg.training.forgetting_threshold`` (including through ``getattr``), the
allowlist-staleness test goes red.
"""
for path in paths:
try:
tree = parse_source(pathlib.Path(path).read_text(errors="ignore"), keep=True)
except (SyntaxError, UnicodeDecodeError, ValueError):
continue
for node in walk_nodes(tree):
if (
isinstance(node, ast.Attribute)
and isinstance(node.ctx, ast.Load)
and node.attr == field
and isinstance(node.value, ast.Attribute)
and node.value.attr == "training"
):
return True
if not isinstance(node, ast.Call) or len(node.args) < 2:
continue
if not isinstance(node.func, ast.Name) or node.func.id not in {"getattr", "hasattr"}:
continue
receiver, name = node.args[:2]
if (
isinstance(receiver, ast.Attribute)
and receiver.attr == "training"
and isinstance(name, ast.Constant)
and name.value == field
):
return True
return False
def schema_property_reads(schema_path: pathlib.Path) -> dict:
"""Fields read inside `schema.py` `@property` bodies, keyed by property name.
`schema.py` is excluded from the main scan, so a field read only through a
resolver there looks like an orphan. `double_quant_on` (schema.py:1880,
#321) is that case: consumers read the property, nothing names the field.
Properties only, never validators, and the distinction is principled: a
validator CHECKS a value and consumes nothing on the user's behalf; a
`@property` RESOLVES one for someone else to consume. Measured on this
schema: 1 property, 156 validators, 0 other decorated functions -- so this
pass contributes exactly one name today.
Returned per-property rather than flattened so the caller can gate on
whether anything actually calls the property. An uncalled resolver must not
launder its fields into "consumed" -- that would reintroduce this
detector's own failure mode one level up.
"""
out: dict = {}
try:
tree = parse_source(schema_path.read_text(errors="ignore"), keep=True)
except (SyntaxError, UnicodeDecodeError, ValueError): # pragma: no cover
return out
for node in walk_nodes(tree):
if not isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
continue
names = []
for dec in node.decorator_list:
if isinstance(dec, ast.Name):
names.append(dec.id)
elif isinstance(dec, ast.Attribute):
names.append(dec.attr)
if "property" not in names:
continue
reads = set()
for inner in ast.walk(node):
if isinstance(inner, ast.Attribute) and isinstance(inner.ctx, ast.Load):
reads.add(inner.attr)
out[node.name] = reads
return out
def fold_property_reads(consumed: set, props: dict) -> set:
"""Add a resolver's reads to `consumed`, but ONLY if the resolver is called.
The gate, extracted so it is testable on its own. Without it a property
nobody calls would launder its fields into "consumed" -- this detector's
own failure mode one level up, where a field is "read" by code that never
runs. Tested against a synthetic uncalled resolver rather than the real
schema, because the real schema's only property IS called, so a test using
it cannot tell the gate from its absence. (An earlier version of that test
did exactly that and the mutation survived.)
"""
out = set(consumed)
for prop_name, reads in props.items():
if prop_name in out:
out |= reads
return out
def _consumer_modules():
return [p for p in SRC.rglob("*.py") if p.name != SCHEMA]
# Fields whose name collides with an unrelated read elsewhere in src/, so only
# a read through a `*.training` receiver counts as consumption.
_RECEIVER_QUALIFIED = frozenset({
"training.forgetting_threshold", # ship.py has a local of the same name
"training.load_in_16bit", # migrate/unsloth.py reads an Unsloth kwarg of that name
})
def field_reaches_a_consumer(key: str, attr: str, consumed: set) -> bool:
"""Apply receiver-qualified checks where the global namespace collides."""
if key in _RECEIVER_QUALIFIED:
return training_receiver_reads(_consumer_modules(), attr)
return attr in consumed
def _consumed_in_src() -> set:
"""Cached: the walk is 499 modules and ~2s, and this file did it four
times uncached. Keyed on the file list so a changed tree re-walks.
Includes fields resolved by a `schema.py` `@property` -- but only when the
property itself is called from somewhere in `src/`. See
`schema_property_reads`.
"""
modules = _consumer_modules()
paths_key = tuple(sorted(str(p) for p in modules))
consumed = set(_consumed_cached(paths_key))
consumed = fold_property_reads(consumed, schema_property_reads(SCHEMA_PATH))
# #807: a read inside a function nothing calls is not consumption, for
# the same reason an uncalled @property launders nothing.
scoped, referenced = _dead_function_inputs(paths_key)
return drop_dead_function_reads(consumed, scoped, referenced)
@memoized_scan
def _dead_function_inputs(paths_key) -> tuple:
"""`(function_scoped_reads, referenced_names)` of one file list, walked once.
`_consumed_in_src()` needs both on every call and this file calls it seven
times, which was fourteen more walks of `src/`. Keyed on the file list, like
`_consumed_cached`, and dropped when the module finishes (`tests/_source_scan`).
Handed out read-only: every caller shares the one mapping.
"""
modules = [pathlib.Path(p) for p in paths_key]
return (
types.MappingProxyType(function_scoped_reads(modules)),
frozenset(referenced_names(modules)),
)
# --------------------------------------------------------------------------
# Fields with no consumer today. Each entry is a promise that someone looked.
#
# Seeded so this lands green; the number normally shrinks as fields are wired.
# A detector repair can expose a pre-existing orphan hidden by a name collision;
# adding that field requires a tracked reason and a deliberate count update.
# --------------------------------------------------------------------------
KNOWN_UNCONSUMED = {
# -- documented with a worked example, applied nowhere. Verified by hand.
"training.lr_groups": "#761 -- warns at load from v0.76, refused as of v0.77; "
"utils/lr_groups.py exports parse_lr_groups() and nothing "
"outside schema.py imports it",
# data.mask_history was here until #761 wired it into data/loss_mask.py.
"training.early_stop_patience": "#761 -- warns at load from v0.76, refused as of "
"v0.77; no early-stop callback reads it",
"training.citation_recall_threshold": "#761 -- warns at load from v0.76, refused as "
"of v0.77; validated by utils/citation_faithful.py, "
"never applied",
# -- found by the read/write fix, and the reason that fix exists. A user
# setting that is OVERRIDDEN rather than merely unread, so the strongest
# kind of member this list has.
"training.grace_codebook": "#808 -- warns at load from v0.76, refused as of v0.77 "
"(the string appears as an artifact-kind name in "
"store.py:52 / edit.py:312, unrelated to this field)",
"training.rl_checkpoint_include_ref_model": "#808 -- warns at load from v0.76, "
"refused as of v0.77; no save writes "
"reference-model state, so nothing reads it",
"training.rl_checkpoint_include_rollout_buffer": "#808 -- warns at load from "
"v0.76, refused as of v0.77; no save "
"writes rollout-buffer state, so "
"nothing reads it",
# -- #807: read ONLY inside a function nothing in src/ references, so the
# read is not consumption. Surfaced by the dead-function gate, which
# the guard previously applied to @property resolvers only.
"training.warmup_auto": "#807 -- read only in autopilot.generate_config, "
"which nothing in src/ calls; and that line reads "
"the decisions dict and WRITES the value into a "
"config, so it is not a read of the field either",
# -- declared, and an explicit value is IGNORED with a warning naming the
# release that refuses it, so having no consumer is correct. Not refused
# yet, because Soup's own writers put the old default into saved configs.
"data.remove_unused_columns": "#759 -- no trainer reads it; the trainers that set the "
"HF argument pass False so a custom collator still sees "
"the extra columns. The default is now False, and an "
"explicit true loads with a warning and is ignored",
# training.packing_cross_doc_attn_mask moved to REFUSED_AT_LOAD (#1814).
# -- staged for features that have not landed; grouped so they can be
# retired together rather than one at a time.
"training.long_context_grpo": "#808 -- warns at load from v0.76, refused as of v0.77 "
"(documented as wiring Tiled MLP; no Tiled MLP exists)",
"training.vision_grpo": "#808 -- warns at load from v0.76, refused as of v0.77 "
"(no vision GRPO path)",
# training.load_in_16bit / unsloth_bnb_4bit / llm_int8 moved to
# VALIDATOR_ALIASES (#1814).
"training.quantize_ref_model": "#808 -- warns at load from v0.76, refused as of v0.77 "
"(reference-model quantisation staging)",
"training.convergence_window": "#808 -- warns at load from v0.76, refused as of v0.77 "
"(convergence-detector staging)",
"training.convergence_rel_tol": "#808 -- warns at load from v0.76, refused as of v0.77 "
"(convergence-detector staging)",
"training.forgetting_eval_steps": "#808 -- warns at load from v0.76, refused as of v0.77 "
"(catastrophic-forgetting probe staging)",
"training.forgetting_threshold": "#799 -- staged catastrophic-forgetting threshold; "
"the same name in ship.py is unrelated",
"training.forgetting_benchmark": "#808 -- warns at load from v0.76, refused as of v0.77 "
"(catastrophic-forgetting probe staging)",
"training.forgetting_stop": "#808 -- warns at load from v0.76, refused as of v0.77 "
"(catastrophic-forgetting probe staging)",
"training.checkpoint_eval_steps": "#808 -- warns at load from v0.76, refused as of v0.77 "
"(checkpoint-eval staging)",
"training.checkpoint_eval_metric": "#808 -- warns at load from v0.76, refused as of v0.77 "
"(checkpoint-eval staging)",
"training.checkpoint_eval_tasks": "#808 -- warns at load from v0.76, refused as of v0.77 "
"(checkpoint-eval staging)",
"training.checkpoint_keep_top": "#808 -- warns at load from v0.76, refused as of v0.77 "
"(checkpoint-eval staging)",
"training.grace_codebook_size": "#808 -- warns at load from v0.76, refused as of v0.77 "
"(GRACE codebook staging)",
"training.grace_codebook_dim": "#808 -- warns at load from v0.76, refused as of v0.77 "
"(GRACE codebook staging)",
"data.video_dir": "#808 -- warns at load from v0.76, refused as of v0.77 "
"(video pipeline staging)",
"data.eval_on_each_dataset": "#808 -- warns at load from v0.76, refused as of v0.77 "
"(per-dataset eval staging)",
"data.split_thinking": "#808 -- warns at load from v0.76, refused as of v0.77 "
"(thinking-block masking staging)",
"data.image_min_pixels": "#808 -- warns at load from v0.76, refused as of v0.77 "
"(image preprocessing staging)",
"data.image_max_pixels": "#808 -- warns at load from v0.76, refused as of v0.77 "
"(image preprocessing staging)",
"data.image_resize_algorithm": "#808 -- warns at load from v0.76, refused as of v0.77 "
"(image preprocessing staging)",
"data.video_fps": "#808 -- warns at load from v0.76, refused as of v0.77 "
"(video pipeline staging)",
"data.video_maxlen": "#808 -- warns at load from v0.76, refused as of v0.77 "
"(video pipeline staging)",
"data.resize_vocab": "#808 -- warns at load from v0.76, refused as of v0.77 "
"(vocab-resize staging)",
"data.extend_conversation": "#808 -- warns at load from v0.76, refused as of v0.77 "
"(conversation-extension staging)",
"data.skip_prepare_dataset": "#808 -- warns at load from v0.76, refused as of v0.77 "
"(dataset-prep bypass staging)",
}
# --------------------------------------------------------------------------
# #1814 -- fields whose only reader is a schema validator. They are not silently
# inert, so they do not belong in KNOWN_UNCONSUMED (whose pin should count only
# fields a user can set to no effect and no word). Each entry names the validator
# that consumes it, and TestTheNamedCategoriesAreTrue checks the behaviour, so an
# entry cannot outlive the validator that justifies it.
# --------------------------------------------------------------------------
#: Declared and refused at config load: setting the field stops the load with a
#: message naming it, which is the strongest kind of "the user is told".
REFUSED_AT_LOAD = {
"training.packing_cross_doc_attn_mask": (
"TrainingConfig._validate_cross_doc_attn_mask",
"#1814 -- refused at config load: it never mapped to a valid TRL "
"packing_strategy (documented in docs/performance-and-quantization.md)",
),
}
#: Aliases a validator consumes: it rewrites another field from them, or asserts
#: that the field they describe is set to match, and refuses the config if not.
VALIDATOR_ALIASES = {
"training.load_in_16bit": (
"TrainingConfig._validate_load_in_aliases",
"#1814 -- rewrites training.quantization to 'none' (and refuses it beside "
"load_in_8bit or a Quant Menu format)",
),
"training.llm_int8": (
"SoupConfig._validate_llm_int8_alias",
"#1814 -- asserts training.quantization == '8bit' and refuses the config otherwise",
),
"training.unsloth_bnb_4bit": (
"SoupConfig._validate_unsloth_bnb_4bit_compat",
"#1814 -- asserts backend 'unsloth' and quantization '4bit' and refuses the "
"config otherwise",
),
}
#: Every field the orphan check lets through, with its reason.
GUARDED = {
**KNOWN_UNCONSUMED,
**{key: reason for key, (_, reason) in REFUSED_AT_LOAD.items()},
**{key: reason for key, (_, reason) in VALIDATOR_ALIASES.items()},
}
def describe_tree_mismatch(imported: pathlib.Path, scanned: pathlib.Path):
"""Return a message if these are different trees, else None.
Compared by **identity, not spelling**. The original form was
``imported == scanned``, which is wrong on a case-insensitive filesystem:
macOS preserves case but does not distinguish it, and ``Path.resolve()``
returns the path as typed. Handing pytest this file as
``/users/.../soup/tests/...`` gives a lowercase ``scanned`` (it is derived
from ``__file__``) while the editable install resolves to the canonical
``/Users/.../Soup``, so two spellings of one directory compared
unequal and the guard reported a broken rebase that had not happened.
A good error message for a condition that is not an error is still a false
alarm.
``os.path.samefile`` compares device and inode, so it answers the question
the assertion is actually asking. It RAISES on a missing path rather than
returning False, which is why existence is checked first -- that is the
one case the assertion most needs to report rather than crash on.
No test-count evidence is quoted here on purpose. The previous comment
cited "17 passed without PYTHONPATH, 2 failed with it", measured against a
17-test file and the ``==`` version, so the number was stale on its own
terms before the comparison changed. `TestTheTreeMismatchCheck` pins the
behaviour instead, which cannot go stale the way a remembered count does.
"""
if not imported.exists():
return f"the imported soup_cli does not exist on disk: {imported}"
if not scanned.exists():
return f"the tree being scanned does not exist: {scanned}"
if os.path.samefile(imported, scanned):
return None
return (
f"scanning {scanned} but importing {imported}; set "
"PYTHONPATH=<checkout>/src or reinstall with `pip install -e .`, "
"or this guard silently passes against the wrong source"
)
def _declared():
import soup_cli
from soup_cli.config.schema import DataConfig, LoraConfig, TrainingConfig
# The scan walks SRC (this checkout); the fields come from the IMPORTED
# package. In a worktree with no PYTHONPATH those are different trees and
# the mismatch fails GREEN -- the silent-pass failure mode this file
# exists to prevent.
imported = pathlib.Path(soup_cli.__file__).resolve().parent
problem = describe_tree_mismatch(imported, SRC)
assert problem is None, problem
out = {}
# #807: LoraConfig was outside the guard's scope entirely, so every
# LoRA-variant field was unguarded.
for cls, label in (
(TrainingConfig, "training"),
(DataConfig, "data"),
(LoraConfig, "training.lora"),
):
for name in cls.model_fields:
out[f"{label}.{name}"] = name
return out
class TestTheDetectorItself:
"""The guard is only worth having if the detector is right in BOTH
directions. A false negative lets a dead field through; a false positive
gets the test deleted.
"""
def _consumed(self, tmp_path, source: str) -> set:
module = tmp_path / "consumer.py"
module.write_text(source)
return consumed_names([module])
def _training_receiver_reads(self, tmp_path, source: str, field: str) -> bool:
module = tmp_path / "receiver_consumer.py"
module.write_text(source)
return training_receiver_reads([module], field)
def test_training_receiver_read_is_field_qualified(self, tmp_path):
source = "def f(cfg):\n return cfg.training.forgetting_threshold\n"
assert self._training_receiver_reads(tmp_path, source, "forgetting_threshold")
def test_training_receiver_getattr_is_field_qualified(self, tmp_path):
source = 'def f(cfg):\n return getattr(cfg.training, "forgetting_threshold", None)\n'
assert self._training_receiver_reads(tmp_path, source, "forgetting_threshold")
def test_unrelated_name_does_not_count_as_training_receiver_read(self, tmp_path):
source = "def f(forgetting_threshold):\n return forgetting_threshold\n"
assert not self._training_receiver_reads(tmp_path, source, "forgetting_threshold")
def test_an_attribute_access_counts_as_consumption(self, tmp_path):
assert "widget" in self._consumed(tmp_path, "def f(cfg):\n return cfg.widget\n")
def test_getattr_with_a_string_counts_as_consumption(self, tmp_path):
"""`getattr(tcfg, "fp8_recipe", ...)` is how utils/v028_features.py
reads many real fields."""
src = 'def f(cfg):\n return getattr(cfg, "widget", None)\n'
assert "widget" in self._consumed(tmp_path, src)
def test_a_dict_lookup_counts_as_consumption(self, tmp_path):
assert "widget" in self._consumed(tmp_path, 'def f(d):\n return d["widget"]\n')
# The docstring fixtures below use the bare field name as the ENTIRE
# docstring. An earlier version wrote prose around it ("Talks about
# widget.") and was vacuous: the collected constant is then that whole
# sentence, never the bare name, so the assertion held whether or not
# stripping happened. Found by mutating the stripper -- disabling it
# survived all three. An exact-match docstring is also the realistic
# shape, since generated documentation often is exactly the field name.
def test_a_module_docstring_does_not_count(self, tmp_path):
"""The distinction a grep gets wrong."""
assert "widget" not in self._consumed(tmp_path, '"""widget"""\n')
def test_a_function_docstring_does_not_count(self, tmp_path):
assert "widget" not in self._consumed(
tmp_path, 'def f():\n """widget"""\n return 1\n'
)
def test_a_class_docstring_does_not_count(self, tmp_path):
assert "widget" not in self._consumed(
tmp_path, 'class C:\n """widget"""\n x = 1\n'
)
def test_prose_mentioning_a_field_is_not_a_read_either(self, tmp_path):
"""The `citation_recall_threshold` shape: named inside a longer
message. Collected as the whole sentence, so it never matches the
field name -- pinned so a future change to how constants are split
cannot start counting prose as consumption."""
src = 'def f():\n raise ValueError("widget must be in [0, 1]")\n'
assert "widget" not in self._consumed(tmp_path, src)
def test_an_unrelated_module_consumes_nothing(self, tmp_path):
"""Reject-everything control: the detector must not report a name that
is simply absent, or every field would look consumed."""
assert "widget" not in self._consumed(tmp_path, "x = 1\n")
def test_a_syntactically_broken_module_is_skipped_not_fatal(self, tmp_path):
"""One unparseable file must not take the guard down."""
bad = tmp_path / "bad.py"
bad.write_text("def (:\n")
assert consumed_names([bad]) == set()
class TestEveryDeclaredFieldReachesAConsumer:
def test_no_new_field_is_declared_without_a_consumer(self):
consumed = _consumed_in_src()
orphans = sorted(
key
for key, attr in _declared().items()
if not field_reaches_a_consumer(key, attr, consumed)
and key not in GUARDED
)
assert not orphans, (
"These config fields are declared in schema.py and read by no "
"module outside it, so a user setting them gets no effect and no "
"warning:\n "
+ "\n ".join(orphans)
+ "\n\nWire the field, or add it to KNOWN_UNCONSUMED with a reason."
)
def test_the_allowlist_names_only_real_fields(self):
"""A renamed or deleted field must not keep a stale entry alive --
otherwise the allowlist silently stops guarding anything."""
declared = _declared()
stale = sorted(k for k in GUARDED if k not in declared)
assert not stale, (
"KNOWN_UNCONSUMED / REFUSED_AT_LOAD / VALIDATOR_ALIASES name fields that no "
"longer exist; remove them:\n "
+ "\n ".join(stale)
)
def test_the_allowlist_does_not_cover_fields_that_are_consumed(self):
"""The list may only shrink. When a field gets wired, its entry has to
go, or the guard stops noticing if the wiring is later removed."""
consumed = _consumed_in_src()
declared = _declared()
now_wired = sorted(
k for k in GUARDED
if k in declared and field_reaches_a_consumer(k, declared[k], consumed)
)
assert not now_wired, (
"These fields now have a consumer, so their allowlist entry "
"is obsolete and must be deleted:\n " + "\n ".join(now_wired)
)
def test_every_allowlist_entry_carries_a_reason(self):
empty = sorted(k for k, v in GUARDED.items() if not v or len(v) < 10)
assert not empty, f"allowlist entries need a reason: {empty}"
def test_the_guard_can_actually_fail(self, tmp_path, monkeypatch):
"""Acceptance criterion 1, demonstrated rather than described.
A guard that has never been observed failing is not yet known to work.
This adds a field to the real TrainingConfig, confirms the check goes
red naming it, then wires a consumer and confirms it goes green.
"""
from soup_cli.config.schema import TrainingConfig
fields = dict(TrainingConfig.model_fields)
fields["totally_unwired_probe"] = fields["max_grad_norm"]
monkeypatch.setattr(TrainingConfig, "model_fields", fields)
consumed = _consumed_in_src()
orphans = [
key for key, attr in _declared().items()
if not field_reaches_a_consumer(key, attr, consumed)
and key not in GUARDED
]
assert "training.totally_unwired_probe" in orphans, (
f"the guard did not flag an unwired field; it reported {orphans}. "
"Membership, not equality: any other orphan present is a separate "
"finding and must not make this read as a failure to detect."
)
# ...and green once something reads it.
wired = tmp_path / "wired.py"
wired.write_text("def f(cfg):\n return cfg.totally_unwired_probe\n")
# Same composition as the real check, property pass included.
consumed_after = _consumed_in_src() | consumed_names([wired])
assert "totally_unwired_probe" in consumed_after
assert not [
key for key, attr in _declared().items()
if not field_reaches_a_consumer(key, attr, consumed_after)
and key not in GUARDED
]
def test_removing_a_fields_last_consumer_is_caught(self, tmp_path):
"""Acceptance criterion 2: the failure fires on the commit that breaks
it, not months later in a user's run."""
declared = _declared()
# `lr` is read all over the trainers; simulate its last consumer going.
assert "training.lr" in declared
only_docstring = tmp_path / "gone.py"
only_docstring.write_text('"""This module used to apply cfg.lr."""\n')
consumed = consumed_names([only_docstring])
assert "lr" not in consumed, (
"a field named only in a docstring must read as unconsumed"
)
def test_the_allowlist_size_is_pinned_exactly():
"""A ratchet that fails in BOTH directions.
`<= N` catches the list growing -- a field allowlisted rather than wired.
It does not catch the list going STALE: wire a field, forget to delete its
entry, and the bound stays green while the allowlist now describes code
that no longer exists. @MakazhanAlpamys flagged that asymmetry on #751,
having watched #756's registry go stale five times in a day for exactly
that reason.
`==` makes both directions a deliberate, reviewable edit to this line.
**The one case only this test can see** -- and the reason it is not
redundant with the test below -- is an entry added for a BRAND-NEW unwired
field. Nothing is stale then, no entry describes code that moved, and the
count is the only signal that a field was allowlisted instead of wired.
`test_the_allowlist_does_not_cover_fields_that_are_consumed` is the other
half: it names WHICH entry went stale, where this one only says the count
moved.
"""
assert len(KNOWN_UNCONSUMED) == 34, (
f"KNOWN_UNCONSUMED is {len(KNOWN_UNCONSUMED)}, pinned at 34. Going UP "
"means a field was allowlisted rather than wired; going DOWN means an "
"entry was retired, which is the good direction -- lower this number "
"in the same commit."
)
def test_the_named_categories_are_pinned_exactly():
"""The same two-way ratchet for the #1814 categories: an entry added to one of
them is a reviewed decision, not a way around the KNOWN_UNCONSUMED pin."""
assert (len(REFUSED_AT_LOAD), len(VALIDATOR_ALIASES)) == (1, 3), (
f"REFUSED_AT_LOAD is {len(REFUSED_AT_LOAD)} and VALIDATOR_ALIASES is "
f"{len(VALIDATOR_ALIASES)}, pinned at 1 and 3; change the pin in the same commit."
)
assert not (set(REFUSED_AT_LOAD) | set(VALIDATOR_ALIASES)) & set(KNOWN_UNCONSUMED)
assert not set(REFUSED_AT_LOAD) & set(VALIDATOR_ALIASES)
class TestTheNamedCategoriesAreTrue:
"""#1814 -- each category entry is checked, not trusted: the named validator
exists, and setting the field has the effect the entry claims. Deleting the
validator, or the check inside it, turns these red."""
_BASE = "base: org/m\ntask: sft\ndata:\n train: d.jsonl\n"
def _load(self, extra: str, backend: str = ""):
from soup_cli.config.loader import load_config_from_string
text = self._BASE + (f"backend: {backend}\n" if backend else "") + extra
return load_config_from_string(text)
@pytest.mark.parametrize("key", sorted({**REFUSED_AT_LOAD, **VALIDATOR_ALIASES}))
def test_the_named_validator_exists(self, key):
from soup_cli.config import schema
qualname = {**REFUSED_AT_LOAD, **VALIDATOR_ALIASES}[key][0]
cls_name, method = qualname.split(".")
assert callable(getattr(getattr(schema, cls_name), method, None)), qualname
def test_packing_cross_doc_attn_mask_is_refused_by_name(self):
with pytest.raises(ValueError, match="packing_cross_doc_attn_mask"):
self._load("training:\n packing_cross_doc_attn_mask: true\n")
def test_load_in_16bit_rewrites_quantization(self):
cfg = self._load("training:\n quantization: 4bit\n load_in_16bit: true\n")
assert cfg.training.quantization == "none"
def test_llm_int8_asserts_8bit(self):
with pytest.raises(ValueError, match="llm_int8"):
self._load("training:\n quantization: 4bit\n llm_int8: true\n")
cfg = self._load("training:\n quantization: 8bit\n llm_int8: true\n")
assert cfg.training.llm_int8 is True
def test_unsloth_bnb_4bit_asserts_backend_and_4bit(self):
with pytest.raises(ValueError, match="unsloth"):
self._load("training:\n quantization: 4bit\n unsloth_bnb_4bit: true\n")
cfg = self._load("training:\n quantization: 4bit\n unsloth_bnb_4bit: true\n",
backend="unsloth")
assert cfg.training.unsloth_bnb_4bit is True
def _reason_is_accountable(reason: str) -> bool:
"""A reason must cite an issue or say in words that none exists.
Extracted so the predicate itself is testable. Loosening a check makes the
suite pass rather than fail, so the only way to pin it is to assert what it
REJECTS -- see `TestTheAccountabilityPredicate`.
"""
return bool(re.search(r"#\d+", reason)) or "no issue" in reason
class TestTheAccountabilityPredicate:
"""`staging` was accepted as a pass and 31 of 40 entries used it. That loose
predicate is what let the `bnb_4bit_use_double_quant` entry through with a
wrong story attached, so what it REJECTS is the part worth pinning."""
def test_a_bare_staging_note_is_not_accountable(self):
assert not _reason_is_accountable("YaRN staging")
def test_prose_with_no_issue_and_no_admission_is_not_accountable(self):
assert not _reason_is_accountable("documented as wiring Tiled MLP")
def test_an_issue_reference_is_accountable(self):
assert _reason_is_accountable("#759 -- fifteen trainers hardcode it")
def test_an_explicit_admission_is_accountable(self):
assert _reason_is_accountable("no issue yet -- nothing imports it")
assert _reason_is_accountable("no issue needed: refused at config load")
def test_every_allowlist_entry_states_an_issue_or_says_there_is_none():
"""An entry with no issue reference is indistinguishable from one someone
added to make CI green, and that is how a ratchet rots. Where no issue
exists the entry must say so out loud, which makes it a standing prompt to
file one -- which is how #759 came to be filed.
"""
# `staging` is deliberately NOT accepted as a pass. 31 of the entries used
# it, and that loose predicate is what let the bnb_4bit_use_double_quant
# entry through carrying a wrong story. An entry must cite an issue or say
# in words that none exists.
vague = sorted(
k for k, v in GUARDED.items()
if not _reason_is_accountable(v)
)
assert not vague, (
"these allowlist entries cite no issue and do not say one is missing:\n "
+ "\n ".join(vague)
)
def test_a_get_default_is_not_counted_as_a_read(tmp_path):
"""`d.get("k", "widget")` returns "widget" as a fallback VALUE, not as a
field name. Counting it would be a false positive, and the false-positive
rate is what decides whether a guard survives the next person in a hurry.
"""
module = tmp_path / "m.py"
module.write_text('def f(d):\n return d.get("k", "widget")\n')
consumed = consumed_names([module])
assert "k" in consumed, "the looked-up key is a read"
assert "widget" not in consumed, "the default value is not a read"
def test_getattr_reads_the_name_not_the_default(tmp_path):
"""`getattr(o, "name", "widget")` -- the name is argument 1, the default 2."""
module = tmp_path / "m.py"
module.write_text('def f(o):\n return getattr(o, "name", "widget")\n')
consumed = consumed_names([module])
assert "name" in consumed
assert "widget" not in consumed
class TestSchemaSideResolvers:
"""`schema.py` is excluded from the scan, so a field read only through a
resolver defined there looked like an orphan. `double_quant_on`
(`schema.py:1880`, #321) is that case, and an earlier version of this file
recorded its field as an unread offender on exactly that evidence.
Properties only, never validators: a validator CHECKS a value and consumes
nothing on the user's behalf; a `@property` RESOLVES one for someone else
to consume. Measured on this schema -- 1 property, 156 validators, 0 other
decorated functions -- so the pass contributes exactly one name today.
"""
def _props(self, tmp_path, source: str) -> dict:
schema = tmp_path / "schema.py"
schema.write_text(source)
return schema_property_reads(schema)
def test_a_property_body_contributes_the_fields_it_reads(self, tmp_path):
src = (
"class C:\n"
" @property\n"
" def resolved(self):\n"
" return self.raw_field is not False\n"
)
assert self._props(tmp_path, src) == {"resolved": {"raw_field"}}
def test_a_validator_body_contributes_nothing(self, tmp_path):
"""156 of them read fields here. Counting those would mark most of the
schema consumed by its own validation, which is the false negative
that matters."""
src = (
"class C:\n"
" @field_validator('raw_field')\n"
" def check(cls, v):\n"
" return cls.raw_field\n"
)
assert self._props(tmp_path, src) == {}
def test_the_real_schema_has_exactly_one_property(self):
"""If a second resolver appears, this pass grows silently -- the pin is
cheap and makes that a deliberate edit."""
props = schema_property_reads(SCHEMA_PATH)
assert list(props) == ["double_quant_on"], (
f"schema.py properties are now {sorted(props)}; each one launders "
"the fields it reads into 'consumed', so review the addition"
)
assert "bnb_4bit_use_double_quant" in props["double_quant_on"]
def test_a_called_resolver_makes_its_field_consumed(self):
"""`double_quant_on` is called from quant_menu.py and stream_setup.py,
so its field retires from the allowlist."""
consumed = _consumed_in_src()
assert "double_quant_on" in consumed, "the property itself must be called"
assert "bnb_4bit_use_double_quant" in consumed
assert "training.bnb_4bit_use_double_quant" not in KNOWN_UNCONSUMED
def test_an_uncalled_resolver_launders_nothing(self):
"""The gate, exercised directly.
An earlier version of this test built a temp schema and then asserted
against `_consumed_in_src()`, which reads the REAL schema -- so it held
whether the gate existed or not, and dropping the gate survived
mutation. This drives `fold_property_reads` itself.
"""
props = {"never_called": {"orphan_field"}, "is_called": {"wired_field"}}
consumed = fold_property_reads({"is_called", "unrelated"}, props)
assert "wired_field" in consumed, "a CALLED resolver contributes its reads"
assert "orphan_field" not in consumed, (
"an uncalled resolver laundered its field into 'consumed'; a field "
"read only by code nothing invokes is not consumed"
)
def test_the_gate_is_what_retires_the_double_quant_entry(self):
"""End to end on the real schema: the property is called, so its field
is consumed and needs no allowlist entry."""
props = schema_property_reads(SCHEMA_PATH)
raw = set(_consumed_cached(tuple(sorted(str(p) for p in _consumer_modules()))))
assert "bnb_4bit_use_double_quant" not in raw, (
"nothing outside schema.py names the field -- that is why the "
"property pass exists"
)
assert "double_quant_on" in raw, "the property itself is called"
assert "bnb_4bit_use_double_quant" in fold_property_reads(raw, props)
def test_the_known_leak_is_still_the_known_leak():
"""Pin the boundary of the global-name-space hole, measured not assumed.
The maintainer measured this on #751 before merging: an unwired
`training.verbose` is caught, an unwired `training.top_k` is not, because
`top_k` appears as an attribute elsewhere in the tree. Recording it as a
test rather than as prose means the hole cannot quietly widen or close.
If a name moves from one list to the other, that is not a failure to fix
by editing this test — it is a change in what the guard can see, and the
docstring's leak table should move with it.
"""
consumed = _consumed_in_src()
assert "verbose" not in consumed, (
"`training.verbose` used to be catchable; if some module now reads a "
"`.verbose` attribute, the guard has lost a name it could see"
)
for leaky in ("top_k", "top_p", "temperature", "dtype", "seed", "logging_steps"):
assert leaky in consumed, (
f"`{leaky}` no longer collides with an unrelated attribute, so the "
"guard can now catch an unwired field of that name. Good news — "
"update the leak table in the module docstring."
)
# --------------------------------------------------------------------------
# #807: the same laundering the property gate refuses, one level over.
#
# `fold_property_reads` gates a `schema.py` @property on being CALLED. Ordinary
# functions got no such gate, so a read inside a function nothing calls counted
# exactly like a read in a trainer. `training.warmup_auto` passed the guard on
# the strength of one line in `autopilot/generate_config.generate_config`,
# which nothing in src/ calls -- and which reads the autopilot decisions dict
# and writes the value INTO a config, so it is not even a read of the field.
# --------------------------------------------------------------------------
def function_scoped_reads(paths) -> dict:
"""Map each read name to the {(module, enclosing function)} it occurs in.
``None`` as the function means module scope, which always counts: a read at
import time runs whenever the module is imported.
Every function is attributed, **methods included**. An earlier version of
this walked only `tree.body` top-level defs, so a read inside a class
method was invisible -- and a field whose only live read sits in a method
then looked dead. That produced two false positives on the real tree
(`auto_mixed_precision`, read at `trainer/sft.py:1185`, and
`training.multipack`). False positives are what get a guard deleted, so the
attribution has to cover methods or the gate is worse than no gate.
"""
out: dict = {}
for path in paths:
try:
tree = parse_source(path.read_text(errors="ignore"), keep=True)
except (SyntaxError, UnicodeDecodeError, ValueError):
continue
stack = [(tree, None)]
while stack:
node, fname = stack.pop()
for child in ast.iter_child_nodes(node):
inner = (
child.name
if isinstance(child, (ast.FunctionDef, ast.AsyncFunctionDef))
else fname
)
name = None
if isinstance(child, ast.Attribute) and isinstance(child.ctx, ast.Load):
name = child.attr
elif isinstance(child, ast.Constant) and isinstance(child.value, str):
name = child.value
if name:
out.setdefault(name, set()).add((path.name, fname))
stack.append((child, inner))
return out
def referenced_names(paths) -> set:
"""Every name loaded anywhere under src/ -- the "is this function called?" set.
A global name match, deliberately. It errs toward "referenced", which is
the safe direction: a guard that cries wolf on live code gets deleted, and
a missed dead function is only a return to today's behaviour.
"""
names: set = set()
for path in paths:
try:
tree = parse_source(path.read_text(errors="ignore"), keep=True)
except (SyntaxError, UnicodeDecodeError, ValueError):
continue
for node in walk_nodes(tree):
if isinstance(node, ast.Name) and isinstance(node.ctx, ast.Load):
names.add(node.id)
elif isinstance(node, ast.Attribute) and isinstance(node.ctx, ast.Load):
names.add(node.attr)
return names
def drop_dead_function_reads(consumed: set, scoped: dict, referenced: set) -> set:
"""Remove names whose EVERY read sits in a function nothing references.
The function-level twin of `fold_property_reads`, and extracted for the
same reason: a gate can only be pinned by driving it directly. Testing it
through the real tree cannot distinguish the gate from its absence for any
field that is genuinely live. Its limits are stated in the module
docstring's "What this cannot see" list.
"""
out = set(consumed)
for name, sites in scoped.items():
if name not in out:
continue
if all(fn is not None and fn not in referenced for _, fn in sites):
out.discard(name)
return out
class TestTheDeadFunctionGate:
"""#807. Both directions, driven through the gate rather than the tree."""
def test_a_read_in_a_referenced_function_still_counts(self):
consumed = {"widget"}
scoped = {"widget": {("m.py", "live_helper")}}
assert "widget" in drop_dead_function_reads(consumed, scoped, {"live_helper"})
def test_a_read_in_a_function_nothing_references_is_dropped(self):
consumed = {"widget"}
scoped = {"widget": {("m.py", "dead_helper")}}
assert "widget" not in drop_dead_function_reads(consumed, scoped, {"other"})
def test_one_live_read_rescues_a_field_with_dead_ones(self):
"""A field read in both a live and a dead function is consumed. Anything
else would flag `auto_mixed_precision`, which `sft.py` really reads."""
consumed = {"widget"}
scoped = {"widget": {("dead.py", "never_called"), ("live.py", "used")}}
assert "widget" in drop_dead_function_reads(consumed, scoped, {"used"})
def test_a_module_scope_read_always_counts(self):
"""Module scope runs on import; there is no enclosing function to be
dead."""
consumed = {"widget"}
assert "widget" in drop_dead_function_reads(
consumed, {"widget": {("m.py", None)}}, set()
)
def test_methods_are_attributed_not_skipped(self, tmp_path):
"""The false-positive generator. Walking only top-level defs made a
read inside a method invisible, so the field looked dead."""
mod = tmp_path / "m.py"
mod.write_text(
"class C:\n"
" def a_method(self, cfg):\n"
" return cfg.widget\n"
)
scoped = function_scoped_reads([mod])
assert scoped.get("widget") == {("m.py", "a_method")}, (
"a read inside a method must be attributed to that method, not lost"
)
def test_the_real_tree_has_exactly_one_known_escape(self):
"""Measured, and pinned so the number cannot drift unnoticed."""
modules = _consumer_modules()
scoped = function_scoped_reads(modules)
referenced = referenced_names(modules)
# The UNGATED set: _consumed_in_src() already applies the gate, so
# diffing it against itself would always be empty and this test would
# pass vacuously. Rebuild the pre-gate set the same way it does.
ungated = fold_property_reads(
set(_consumed_cached(tuple(sorted(str(m) for m in modules)))),
schema_property_reads(SCHEMA_PATH),
)
dropped = sorted(ungated - drop_dead_function_reads(ungated, scoped, referenced))
assert "warmup_auto" in dropped, (
"warmup_auto is read only inside a function nothing references, "
"so the gate should drop it"
)
# The name says EXACTLY one, and membership alone would pass a gate that
# dropped more (#906 review). `dropped` spans every consumed identifier,
# not just config fields, so the exact claim is over declared leaves.
declared_leaves = set(_declared().values())
dropped_fields = sorted(set(dropped) & declared_leaves)
assert dropped_fields == ["warmup_auto"], (
f"expected exactly the one known escape, got {dropped_fields}"
)
def test_consumed_in_src_applies_the_gate_independently_of_the_allowlist(self):
"""#906 review: the only thing that caught the gate being removed from
`_consumed_in_src()` was `test_the_allowlist_does_not_cover_fields_that_
are_consumed` -- the pin scheduled for deletion once #794/#808 retire
those entries. This pins the composition directly: each escape is read
in the ungated set, and must be absent from what the scan returns."""
modules = _consumer_modules()
ungated = fold_property_reads(
set(_consumed_cached(tuple(sorted(str(m) for m in modules)))),
schema_property_reads(SCHEMA_PATH),
)
consumed = _consumed_in_src()
assert "warmup_auto" in ungated, (
"warmup_auto is not read at all, so this check is vacuous"
)
assert "warmup_auto" not in consumed, (
"warmup_auto is read only inside a dead function, yet _consumed_in_src() "
"counts it: the gate is not composed into the scan"
)
class TestTheTreeMismatchCheck:
"""The scan/import guard, compared by identity rather than by spelling.
Reported by another session on this account: given this file by a
lowercased path, the guard failed with a message that reads like a broken
rebase. A lowercased `cd` alone does not do it -- `getcwd()` returns the
canonical spelling -- so the spelling arrives through `__file__`. macOS is
case-insensitive but case-PRESERVING and `Path.resolve()` does not
normalise case, so `==` compared two spellings of one directory and called
them different.
These replace a comment that cited "17 passed without PYTHONPATH, 2 failed
with it" as its evidence. That count was measured against the `==` version
and against a 17-test file that has grown since. A number that travels
between implementations is exactly what a test should be doing instead.
"""
@pytest.mark.requires_symlink
def test_the_same_directory_reached_by_a_different_spelling_is_not_a_mismatch(
self, tmp_path
):
"""The false alarm, reproduced through a symlink rather than by relying
on the filesystem being case-insensitive -- so this test means the same
thing on Linux CI, where `/Users` and `/users` really are different."""
real = tmp_path / "Soup" / "src" / "soup_cli"
real.mkdir(parents=True)
alias = tmp_path / "alias"
alias.symlink_to(tmp_path / "Soup")
other_spelling = alias / "src" / "soup_cli"
assert real != other_spelling, "the two spellings differ as strings"
assert describe_tree_mismatch(real, other_spelling) is None, (
"one directory reached two ways is not a mismatch; comparing "
"spelling rather than identity is what produced the false alarm"
)
def test_genuinely_different_trees_are_still_caught(self, tmp_path):
"""The case the assertion exists for, and the one that must not be lost
to the fix: a worktree with no PYTHONPATH scans one tree while
importing another, and the guard would otherwise pass GREEN."""
a = tmp_path / "checkout" / "src" / "soup_cli"
b = tmp_path / "installed" / "soup_cli"
a.mkdir(parents=True)
b.mkdir(parents=True)
msg = describe_tree_mismatch(b, a)
assert msg is not None
assert "PYTHONPATH" in msg, "the message must say how to fix it"
def test_a_missing_imported_path_is_reported_not_crashed(self, tmp_path):
"""`os.path.samefile` raises rather than returning False on a missing
path, so existence is checked first. This is the case the assertion
most needs to report."""
scanned = tmp_path / "src" / "soup_cli"
scanned.mkdir(parents=True)
msg = describe_tree_mismatch(tmp_path / "gone", scanned)
assert msg is not None and "does not exist" in msg
def test_a_missing_scanned_path_is_reported_not_crashed(self, tmp_path):
imported = tmp_path / "soup_cli"
imported.mkdir()
msg = describe_tree_mismatch(imported, tmp_path / "gone")
assert msg is not None and "does not exist" in msg
def test_the_real_checkout_agrees_with_itself(self):
"""Control on the live tree: whatever this run's spelling, the guard
must not fire."""
import soup_cli
imported = pathlib.Path(soup_cli.__file__).resolve().parent
assert describe_tree_mismatch(imported, SRC) is None