Skip to content

armgit / Soup

Public Imported from GitHub
0

Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.

45.5 KB
Raw Blame
"""v0.73.0 (#331) — the repair: keep streamed NF4 weights out of ``MatMul4Bit``.

WHY THIS EXISTS

``bitsandbytes.autograd._functions.MatMul4Bit.forward`` stashes the packed weight
and the ``quant_state`` on ``ctx`` as PLAIN ATTRIBUTES::

    ctx.state = quant_state
    ctx.tensors = (None, B)

They never go through ``save_for_backward``, so ``torch.utils.checkpoint`` cannot
discard and recompute them. The reference is taken in the forward, it ALIASES the
streaming buffer pool, and it is read in the backward after that slot has already
been refilled with a different layer.

Symptom, measured on 8xH100 against a resident NF4 reference (see
``benchmarks/gate-h100-validation.md``): the forward stays bit-exact, the loss curve
looks healthy, and the gradients are wrong on every layer except the last
``stream_buffers``. It bites NF4 above roughly 165 MiB per layer, so 32B and 72B, and
never bf16 — which goes through ``MmBackward0`` with a normal ``save_for_backward``.

Both de-aliasing repairs were measured and rejected: because bnb holds the reference
across the whole forward-to-backward span, ANY de-aliasing keeps one copy of every
layer alive for that span and costs O(model), not O(window). On real 32B that was
peak VRAM 4 220 -> 19 720 MiB, which deletes the feature's premise.

THE REPAIR: do not send a streamed NF4 weight through bitsandbytes'
``MatMul4Bit`` autograd Function at all. v0.73.0 dequantised inside the
checkpointed region and used ``F.linear``. #842 may instead call the native
``bitsandbytes::gemm_4bit`` forward through Soup's own autograd Function, whose
packed weight and quantisation tensors all pass through ``save_for_backward``.
Either route is visible to checkpointing; the unsafe plain-ctx lifetime remains
forbidden.

HOW #842 PRESERVES THE NATIVE BITSANDBYTES REFERENCE

``bitsandbytes::gemm_4bit`` chooses between its fused kernel and
``_dequant_linear_fallback``. The constant this file used to cite is only the OUTER
bound of that choice::

    _gemm_4bit_custom_max_m = 1536      # CUDA (256 on ROCm)
    if M > _gemm_4bit_custom_max_m:  -> _dequant_linear_fallback
    elif K % blocksize != 0:         -> _dequant_linear_fallback
    else: use_custom = _gemm_4bit_use_custom_fn(device, dtype, M, N, K)

Below that bound the decision is NOT a threshold on M at all: it is a per-arch
occupancy heuristic over the full shape ``(M, N, K)`` and the device. The constant
therefore explains the ``M > 1536`` half only, and citing it alone is not an argument
about training shapes.

Soup now asks those private bitsandbytes dispatch symbols the same question. At a
custom-GEMM shape it calls that GEMM through a checkpoint-visible Function; at a
fallback shape it retains the dequantise + ``F.linear`` repair. Thus the streamed
model matches an unpatched resident bitsandbytes model at both shapes without
paying the Function's extra backward dequantisation at ordinary training M. If the
private symbols move, Soup fails toward the safe Function and a GPU test reports
the upstream change.
"""

import os
import sys
import types

import pytest

# The NF4 streamed/resident pair builders live in test_v07202 and are deliberately
# NOT duplicated here: two copies of a fixture drift, and this file's whole subject
# is a numerical comparison between the two models they build.
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))

torch = pytest.importorskip("torch")
pytest.importorskip("bitsandbytes")

from test_v07202 import _nf4_stream, _resident_nf4  # noqa: E402


def _count_matmul_4bit(monkeypatch):
    """Count ``bnb.matmul_4bit`` calls.

    ``bitsandbytes/nn/modules.py`` does ``import bitsandbytes as bnb`` and then calls
    ``bnb.matmul_4bit(...)``, i.e. the attribute is looked up on the module object at
    CALL time, so patching the module attribute intercepts it.
    """
    import bitsandbytes as bnb

    calls = {"n": 0}
    real = bnb.matmul_4bit

    def counting(*args, **kwargs):
        calls["n"] += 1
        return real(*args, **kwargs)

    monkeypatch.setattr(bnb, "matmul_4bit", counting)
    return calls


class TestStreamedNF4AvoidsMatMul4Bit:
    """The repair, and the control that makes it mean something.

    ``0 calls`` on its own is equally consistent with "the counter never
    intercepted anything" — which is exactly how an earlier path control in this
    investigation was fooled. The resident control must COUNT, in the same test
    session, or the streamed assertion proves nothing.
    """

    def test_resident_nf4_does_reach_matmul_4bit(self, tmp_path, monkeypatch):
        """CONTROL. Without this, the assertion below is unfalsifiable."""
        _, _, weights, _, _ = _nf4_stream(tmp_path)
        resident = _resident_nf4(weights)
        calls = _count_matmul_4bit(monkeypatch)
        with torch.no_grad():
            resident(input_ids=torch.randint(0, 64, (1, 8)))
        assert calls["n"] > 0, (
            "the counter did not intercept bnb.matmul_4bit at all, so it cannot "
            "detect the streamed path avoiding it either"
        )

    def test_streamed_nf4_forward_does_not_reach_matmul_4bit(self, tmp_path, monkeypatch):
        model, _, _, _, _ = _nf4_stream(tmp_path)
        calls = _count_matmul_4bit(monkeypatch)
        with torch.no_grad():
            model(input_ids=torch.randint(0, 64, (1, 8)))
        assert calls["n"] == 0, (
            f"streamed NF4 still routed {calls['n']} call(s) through MatMul4Bit, which "
            "captures the packed weight outside save_for_backward and therefore aliases "
            "the buffer pool across the checkpoint boundary (#331)"
        )


# ==========================================================================
# #328 — HF gradient checkpointing must stay OFF on a streamed preference run
# ==========================================================================
PREFERENCE_TASKS = ("dpo", "orpo", "simpo", "kto")


class TestStreamedPreferenceLossesDisableHfGradientCheckpointing:
    """``StreamedDecoderLayer`` already wraps every layer in
    ``checkpoint(use_reentrant=False)``. ``should_enable_hf_gradient_checkpointing``
    exists so the HF Trainer does not ALSO checkpoint a streamed model — but only
    ``sft.py`` ever consulted it. The four preference wrappers never did, and TRL's
    ``DPOConfig`` / ``ORPOConfig`` / ``CPOConfig`` / ``KTOConfig`` all default
    ``gradient_checkpointing=True``, so it arrived switched on through the default
    rather than through anything the user wrote.

    On torch 2.5.1 that is the documented ~1.5x silent double-recompute. On torch
    2.13 + CUDA it is fatal: HF's checkpoint wraps the INNER ``LlamaDecoderLayer``,
    whose recompute runs long after ``functional_call``'s reparametrisation context
    has exited and restored the ``meta`` placeholders, so the recompute multiplies a
    ``meta`` ``input_layernorm.weight`` by a real ``cuda`` activation::

        RuntimeError: Tensor on device cuda:0 is not on the expected device meta!

    Measured on 8xH100: all four preference losses fail, SFT passes, and forcing the
    flag back off makes the DPO failure disappear with the wrapper's own checkpoint
    still active — which is what makes HF's checkpointing the cause rather than a
    correlate.
    """

    @pytest.mark.parametrize("task", PREFERENCE_TASKS)
    def test_streaming_turns_hf_gradient_checkpointing_off(self, task, tmp_path, monkeypatch):
        from test_v07204 import _build_streamed_wrapper

        wrapper, _, _ = _build_streamed_wrapper(
            tmp_path, monkeypatch, task=task, device="cpu"
        )
        assert wrapper.trainer.args.gradient_checkpointing is False, (
            f"{task}: HF gradient checkpointing reached the Trainer switched ON for a "
            "streamed run, so HF checkpoints the inner decoder layer as well as the "
            "streaming wrapper. On torch 2.13 CUDA that recompute sees the restored "
            "meta placeholders and dies with 'expected device meta' (#328)"
        )

    @pytest.mark.parametrize("asked", [True, False])
    def test_a_non_streaming_run_gets_what_the_config_asked_for(
        self, asked, tmp_path, monkeypatch
    ):
        """CONTROL, and the second half of the same defect.

        Two things have to be true at once, and only asserting one of them would
        hide the other:

        * the repair must switch HF checkpointing off ONLY where streaming already
          checkpoints — pinning the flag to a constant ``False`` would satisfy the
          streamed test above while silently removing checkpointing from every
          ordinary DPO run;
        * the wrapper must not inherit TRL's default either way.

        Stating it as "the config value survives to the Trainer" is deliberately
        version-independent. TRL's own default for these configs is NOT stable —
        measured ``False`` on trl 0.19.1 and ``True`` on trl 0.26.2 — so asserting
        the default directly would be red on one supported stack and green on the
        other while the actual bug (the wrapper never setting it) went unnoticed on
        both. On the older stack a user's explicit ``gradient_checkpointing: true``
        was being silently dropped; on the newer one TRL's ``True`` arrived
        uninvited. One cause, two opposite symptoms.
        """
        from test_v07204 import _build_streamed_wrapper

        wrapper, _, _ = _build_streamed_wrapper(
            tmp_path,
            monkeypatch,
            task="dpo",
            device="cpu",
            stream_layers=False,
            gradient_checkpointing=asked,
        )
        assert wrapper.trainer.args.gradient_checkpointing is asked


# ==========================================================================
# the CI fixture must exercise the kernel path production actually takes
# ==========================================================================
@pytest.mark.gpu(reason="the fused-kernel window is a CUDA dispatch")
class TestBitsandbytesDispatchAtTrainingShape:
    """The larger fixture shape pins bitsandbytes' fallback branch.

    ``bitsandbytes::gemm_4bit`` dispatches on M. Measured on this fixture (hidden 64,
    so every projection is [64x64] or [32x64]), on an H100:

    ==========  ==========================  ================
    tokens      resident calls via fallback  streamed-vs-resident
    ==========  ==========================  ================
    8, 16, 32   0 of 14 — all FUSED          3.906250e-03 (= 2^-8, one bf16 ulp)
    64 and up   14 of 14 — all fallback      0.0
    ==========  ==========================  ================

    Real 8B / 32B projections take the fallback at every M from 8 to 2048, so the
    fused branch is one the shipped models never reach — and the gate was sitting in
    it, at 16 tokens. The window boundary and the bit-exactness boundary were
    measured to coincide EXACTLY at 64, which is what identifies the dispatch as the
    cause of the ulp rather than something else that merely correlates with size.

    #842 separately pins a fused shape and requires exact parity with an unpatched
    resident model. This test keeps a representative training M on the native
    fallback branch, where Soup must avoid its custom Function.
    """

    def test_every_resident_4bit_call_takes_the_dequant_fallback(self, tmp_path):
        import bitsandbytes as bnb

        bnb_ops = pytest.importorskip(
            "bitsandbytes.backends.cuda.ops",
            reason="the M-dispatch lives in the CUDA backend module",
        )
        if not hasattr(bnb_ops, "_dequant_linear_fallback"):
            pytest.skip(
                "this bitsandbytes has no _dequant_linear_fallback (added around "
                "0.50); the branch it names cannot be counted here. The property "
                "still holds — TestNF4ParityOnCuda asserts the observable "
                "consequence, an exactly-zero streamed-vs-resident difference."
            )

        model, _, weights, _, _ = _nf4_stream(tmp_path, device="cuda", dtype="bfloat16")
        resident = _resident_nf4(weights, dtype="bfloat16", device=0)
        del model

        counts = {"fallback": 0, "total": 0}
        real_fb = bnb_ops._dequant_linear_fallback
        real_matmul = bnb.matmul_4bit

        def counting_fb(*args, **kwargs):
            counts["fallback"] += 1
            return real_fb(*args, **kwargs)

        def counting_matmul(*args, **kwargs):
            counts["total"] += 1
            return real_matmul(*args, **kwargs)

        ids = torch.randint(
            0, 64, (1, 128), generator=torch.Generator().manual_seed(11)
        ).cuda()
        bnb_ops._dequant_linear_fallback = counting_fb
        bnb.matmul_4bit = counting_matmul
        try:
            with torch.no_grad():
                resident(input_ids=ids, attention_mask=torch.ones_like(ids))
            torch.cuda.synchronize()
        finally:
            bnb_ops._dequant_linear_fallback = real_fb
            bnb.matmul_4bit = real_matmul

        assert counts["total"] > 0, (
            "the counter never intercepted bnb.matmul_4bit, so it cannot tell which "
            "branch was taken either"
        )
        assert counts["fallback"] == counts["total"], (
            f"{counts['total'] - counts['fallback']} of {counts['total']} resident "
            "4-bit calls took the custom GEMM at the training-shape control. "
            "Update the #842 fused/fallback matrix for this bitsandbytes dispatch."
        )


# ==========================================================================
# #78 — two defects the FlashAttention / Liger benchmark surfaced
# ==========================================================================
def _sft_wrapper(tmp_path, monkeypatch, **training):
    """A NON-streaming SFT wrapper built through the real ``setup()`` path.

    Deliberately not reusing ``_build_streamed_wrapper``: both defects below are
    about what reaches TRL's ``SFTConfig`` on an ORDINARY run, which is what every
    user gets, and routing through the streaming helper would test a narrower path
    than the one that is broken.
    """
    import yaml
    from test_v07202 import _tiny_llama_dir, _write_tiny_tokenizer

    from soup_cli.config.loader import load_config_from_string
    from soup_cli.trainer.sft import SFTTrainerWrapper

    weights, _, _ = _tiny_llama_dir(tmp_path)
    _write_tiny_tokenizer(weights)
    monkeypatch.chdir(tmp_path)
    max_length = training.pop("max_length", 64)
    tcfg = {
        "batch_size": 1,
        "quantization": "none",
        "epochs": 1,
        "logging_steps": 1,
        "save_steps": 1000,
        "lora": {"r": 4, "alpha": 8, "target_modules": ["q_proj", "v_proj"]},
    }
    tcfg.update(training)
    cfg = load_config_from_string(
        yaml.safe_dump(
            {
                "base": weights,
                "task": "sft",
                "backend": "transformers",
                "modality": "text",
                "data": {
                    "train": "train.jsonl",
                    "max_length": max_length,
                    "chat_template": "chatml",
                },
                "training": tcfg,
                "output": str(tmp_path / "out"),
            }
        )
    )
    wrapper = SFTTrainerWrapper(cfg, device="cpu")
    wrapper.setup({"train": [{"messages": [
        {"role": "user", "content": "hi"},
        {"role": "assistant", "content": "yo"},
    ]}] * 4})
    return wrapper


class TestDataMaxLengthReachesTrl:
    """``data.max_length`` above 1024 was silently ignored on every SFT run.

    Soup builds a ``transformers.TrainingArguments``, not a ``trl.SFTConfig``.
    ``SFTTrainer.__init__`` converts one to the other with
    ``SFTConfig(**args.to_dict())``, and ``max_length`` is an SFT-specific field
    that ``TrainingArguments`` does not carry — so it took ``SFTConfig``'s own
    default of 1024 no matter what the config said.

    Measured on 8xH100 before the fix: ``data.max_length=512`` gave 512 tokens per
    sample, ``data.max_length=4096`` gave **1024**. Every long-context SFT run was
    quietly truncated, and no warning was emitted. It also caps how much
    FlashAttention can ever help, since its advantage grows with sequence length.
    """

    def test_max_length_above_the_trl_default_survives(self, tmp_path, monkeypatch):
        wrapper = _sft_wrapper(tmp_path, monkeypatch, max_length=2048)
        assert wrapper.trainer.args.max_length == 2048, (
            f"TRL received max_length={wrapper.trainer.args.max_length} where the "
            "config asked for 2048 — sequences are being truncated silently"
        )

    def test_max_length_below_the_trl_default_survives_too(self, tmp_path, monkeypatch):
        """CONTROL. A fix that pinned max_length to some large constant would pass
        the test above while breaking every short-sequence run."""
        wrapper = _sft_wrapper(tmp_path, monkeypatch, max_length=128)
        assert wrapper.trainer.args.max_length == 128


class TestLigerTellsTrlItIsOn:
    """``training.use_liger: true`` crashed at step 0 — the feature never ran.

    Soup patches the model with ``apply_liger_kernel()`` but never set
    ``TrainingArguments.use_liger_kernel``, which is the flag TRL and HF read to
    know the fused path will return ``logits=None``. TRL guards its entropy metric
    with ``if not self.args.use_liger_kernel:``; with the flag left False the guard
    cannot fire and ``entropy_from_logits(None)`` raises
    ``AttributeError: 'NoneType' object has no attribute 'shape'``.

    Reproduced on trl 0.26.2 AND trl 0.19.1 — i.e. across the whole supported pin,
    so this was never version-specific bad luck.
    """

    def test_flag_is_set_when_liger_is_requested(self, tmp_path, monkeypatch):
        pytest.importorskip("liger_kernel", reason="use_liger needs the liger extra")
        wrapper = _sft_wrapper(tmp_path, monkeypatch, use_liger=True)
        assert wrapper.trainer.args.use_liger_kernel is True

    def test_flag_stays_off_when_liger_is_not_requested(self, tmp_path, monkeypatch):
        """CONTROL. Setting it unconditionally would make TRL expect None logits
        from an unpatched model, which is the same crash with the arms swapped."""
        wrapper = _sft_wrapper(tmp_path, monkeypatch)
        assert wrapper.trainer.args.use_liger_kernel is False


# ==========================================================================
# #76 — SGLang generation was broken for every request
# ==========================================================================
class TestSglangDecodesItsRuntimeResponse:
    """`--backend sglang` loaded and then returned 500 on every generation.

    ``sglang.Runtime.generate`` ends with ``return json.dumps(response.json())`` —
    a **string**. ``utils/sglang.py`` did ``response["text"]`` on it, so every
    request raised ``TypeError: string indices must be integers``. Deterministic,
    not a race, and the backend had genuinely never been run: verified on
    sglang 0.5.16 on an H100.

    Older sglang returned a dict, so the decode has to accept BOTH shapes rather
    than swapping one hard assumption for another.
    """

    def test_a_json_string_is_decoded(self):
        import json

        from soup_cli.utils.sglang import decode_sglang_response

        payload = {"text": " Paris.", "meta_info": {"prompt_tokens": 17}}
        out = decode_sglang_response(json.dumps(payload))
        assert out["text"] == " Paris."
        assert out["meta_info"]["prompt_tokens"] == 17

    def test_a_dict_still_works(self):
        """CONTROL. The older sglang shape must keep working — a fix that only
        handled strings would break every previously-working version."""
        from soup_cli.utils.sglang import decode_sglang_response

        payload = {"text": " Paris.", "meta_info": {"prompt_tokens": 17}}
        assert decode_sglang_response(payload) is payload

    def test_an_undecodable_response_raises_something_actionable(self):
        from soup_cli.utils.sglang import decode_sglang_response

        with pytest.raises(ValueError, match="SGLang"):
            decode_sglang_response("not json at all")


# ==========================================================================
# #77 — `soup train --gpus N` never launched at all
# ==========================================================================
class TestMultiGpuLaunchArgvIsRunnable:
    """The whole multi-GPU entry point was dead.

    ``accelerate launch`` takes a **script path** (or ``--module NAME``), and Soup
    handed it ``sys.executable``. accelerate then opened the Python ELF binary and
    parsed it as source, so every rank died before the trainer existed::

        File "/root/venv/bin/python", line 1
          ELF
        SyntaxError: source code cannot contain null bytes

    Measured on 4xH100: every arm of the #77 matrix had to be launched by hand.
    The failure is total and immediate, which is why it is worth a test that reads
    the argv rather than one that mocks the launch.
    """

    def test_the_python_dash_m_form_becomes_accelerates_module_flag(self):
        import sys as _sys

        from soup_cli.utils.launcher import build_accelerate_argv

        argv = build_accelerate_argv(
            num_processes=4,
            script_args=[_sys.executable, "-m", "soup_cli.cli", "train", "-c", "s.yaml"],
        )
        assert _sys.executable not in argv, (
            "the Python interpreter is still being passed where accelerate expects "
            "a training script; accelerate will parse the ELF binary as source"
        )
        assert "--module" in argv
        assert argv[argv.index("--module") + 1] == "soup_cli.cli"
        assert argv[-3:] == ["train", "-c", "s.yaml"]
        assert argv[:2] == ["accelerate", "launch"]

    def test_a_real_script_path_is_left_alone(self):
        """CONTROL. Translating unconditionally would break the documented
        ``accelerate launch train.py`` form, which is a script path and must stay
        positional."""
        from soup_cli.utils.launcher import build_accelerate_argv

        argv = build_accelerate_argv(
            num_processes=2, script_args=["train.py", "--config", "s.yaml"]
        )
        assert "--module" not in argv
        assert argv[-3:] == ["train.py", "--config", "s.yaml"]

    def test_single_process_is_still_directly_executable(self):
        """CONTROL. At one process there is no accelerate wrapper and the argv is
        exec'd as-is, so the interpreter form must survive untouched — turning it
        into ``--module`` there would produce a command with no program to run."""
        import sys as _sys

        from soup_cli.utils.launcher import build_accelerate_argv

        argv = build_accelerate_argv(
            num_processes=1, script_args=[_sys.executable, "-m", "soup_cli.cli", "train"]
        )
        assert argv == [_sys.executable, "-m", "soup_cli.cli", "train"]


# ==========================================================================
# #335 — use_fsdp2_compile wrote an adapter that reloads as all zeros
# ==========================================================================
def _write_adapter(directory, prefix=""):
    """Write a two-tensor LoRA adapter, optionally through torch.compile's prefix.

    Module-level so #351's checkpoint tests below build the file the same way
    #335's do. Both are assertions about one exact key spelling, and two copies
    of the fixture that produces it would drift.
    """
    from safetensors.torch import save_file

    directory.mkdir(parents=True, exist_ok=True)
    tensors = {
        f"{prefix}base_model.model.model.layers.0.self_attn.q_proj.lora_A.weight":
            torch.ones(4, 8),
        f"{prefix}base_model.model.model.layers.0.self_attn.q_proj.lora_B.weight":
            torch.full((8, 4), 0.007),
    }
    path = directory / "adapter_model.safetensors"
    save_file(tensors, str(path))
    return path


class TestCompiledAdapterKeysAreCanonical:
    """A run that completes, exits 0, and writes a file that silently does nothing.

    Under ``training.use_fsdp2_compile`` every saved key carried torch.compile's
    wrapper prefix::

        _orig_mod.base_model.model...     # written with compile
        base_model.model...               # written without it

    The tensors ARE trained (max|lora_B| 7.0e-3 measured), but
    ``PeftModel.from_pretrained`` matches none of them, emits only
    ``UserWarning: Found missing adapter keys``, and leaves ``lora_B`` at its zero
    init — **0 of 96 non-zero**, reproduced 3/3 on 4xH100, against 96/96 for the
    paired non-compile run.

    This is the same failure shape v0.72.1 fixed for the streaming wrapper's
    ``.inner.`` segment, and it is the worst one this project has: nothing raises.
    """

    def test_the_compile_prefix_is_stripped(self, tmp_path):
        from safetensors.torch import load_file

        from soup_cli.utils.peft_wiring import strip_compile_prefix

        path = _write_adapter(tmp_path / "a", prefix="_orig_mod.")
        changed = strip_compile_prefix(str(tmp_path / "a"))
        assert changed == 2, f"expected both keys rewritten, got {changed}"
        keys = set(load_file(str(path)))
        assert not any(k.startswith("_orig_mod.") for k in keys), keys
        assert all(k.startswith("base_model.model.") for k in keys), keys

    def test_the_values_survive_the_rewrite(self, tmp_path):
        """Renaming must not quietly re-initialise anything — the whole defect is
        that trained numbers stop being found, so a fix that loses them is the
        same bug with a different mechanism."""
        from safetensors.torch import load_file

        from soup_cli.utils.peft_wiring import strip_compile_prefix

        path = _write_adapter(tmp_path / "a", prefix="_orig_mod.")
        strip_compile_prefix(str(tmp_path / "a"))
        loaded = load_file(str(path))
        b = loaded["base_model.model.model.layers.0.self_attn.q_proj.lora_B.weight"]
        assert float(b.abs().max()) == pytest.approx(0.007), "trained values lost"

    def test_a_normal_adapter_is_untouched(self, tmp_path):
        """CONTROL. Rewriting unconditionally would churn every ordinary run's
        adapter file for nothing, and any bug in the rewrite would then reach
        users who never enabled compile."""
        from safetensors.torch import load_file

        from soup_cli.utils.peft_wiring import strip_compile_prefix

        path = _write_adapter(tmp_path / "a")
        before = set(load_file(str(path)))
        assert strip_compile_prefix(str(tmp_path / "a")) == 0
        assert set(load_file(str(path))) == before

    def test_a_missing_adapter_file_is_not_an_error(self, tmp_path):
        """Full fine-tuning writes no adapter; the normaliser must not turn that
        into a crash at the end of a completed run."""
        from soup_cli.utils.peft_wiring import strip_compile_prefix

        (tmp_path / "empty").mkdir()
        assert strip_compile_prefix(str(tmp_path / "empty")) == 0


# ==========================================================================
# #351: the repair above ran on the final save only
# ==========================================================================
def _fire_on_save(callback, output_dir, step, *, should_save=True, world_zero=None):
    """Drive one ``on_save`` the way HF's CallbackHandler does.

    ``args`` / ``state`` are the two attributes the callback reads, so a
    SimpleNamespace is the whole Trainer this needs, the same shape
    ``tests/test_hf_integration.py`` uses to drive ``HFPushCallback.on_save``.

    ``world_zero`` defaults to following ``should_save``, which is what every
    run Soup can produce today does. Pass it explicitly to drive them apart, the
    way ``save_on_each_node`` does.
    """
    args = types.SimpleNamespace(output_dir=str(output_dir), should_save=should_save)
    state = types.SimpleNamespace(
        global_step=step,
        is_world_process_zero=should_save if world_zero is None else world_zero,
    )
    control = types.SimpleNamespace()
    callback.on_save(args, state, control)


class TestCheckpointAdaptersAreCanonicalToo:
    """#335's repair fires once, after the final ``save_model``. The
    ``checkpoint-*`` directories written on the way there keep the prefix.

    Both are written by the SAME ``Trainer.save_model`` (``_save_checkpoint``
    calls it with ``output_dir=<run>/checkpoint-N``), so they come out identical
    and only the last one was ever normalised. Measured at 70B on 8xH100
    (``benchmarks/gate-h100-validation.md``, STEP 28): 320 canonical keys in the
    output root, 320 prefixed ones in ``checkpoint-100``.

    Resuming is the case that decides how bad this is. ``from_pretrained`` at
    least warns. ``Trainer._load_from_checkpoint`` calls
    ``model.load_adapter(...)`` and drops the return value, and ``load_adapter``
    deliberately does not warn: it hands the missing keys back in the load
    result instead, which nothing reads. The unexpected ``_orig_mod.`` keys go
    to ``load_state_dict(strict=False)``, which discards them without a word. So
    a resumed run continues from a re-zeroed ``lora_B`` in total silence: the
    #335 failure shape with the last warning removed.
    """

    def test_a_checkpoint_written_under_compile_is_normalised(self, tmp_path):
        from safetensors.torch import load_file

        from soup_cli.utils.peft_wiring import build_compile_prefix_callback

        path = _write_adapter(tmp_path / "checkpoint-100", prefix="_orig_mod.")
        _fire_on_save(build_compile_prefix_callback(), tmp_path, 100)

        keys = set(load_file(str(path)))
        assert not any(k.startswith("_orig_mod.") for k in keys), keys
        assert all(k.startswith("base_model.model.") for k in keys), keys

    def test_the_trained_values_survive_in_a_checkpoint(self, tmp_path):
        """The standard #335's repair was held to. A rename that loses the
        numbers is the same defect wearing a different mechanism."""
        from safetensors.torch import load_file

        from soup_cli.utils.peft_wiring import build_compile_prefix_callback

        path = _write_adapter(tmp_path / "checkpoint-100", prefix="_orig_mod.")
        _fire_on_save(build_compile_prefix_callback(), tmp_path, 100)

        loaded = load_file(str(path))
        b = loaded["base_model.model.model.layers.0.self_attn.q_proj.lora_B.weight"]
        assert float(b.abs().max()) == pytest.approx(0.007), "trained values lost"

    def test_a_repaired_checkpoint_is_byte_identical_to_an_uncompiled_one(
        self, tmp_path
    ):
        """What ``resume_from_checkpoint`` actually needs.

        A real resume needs FSDP2 and several GPUs, but the thing being resumed
        is a file, and peft matches it by name. So the assertion that carries
        the same weight off-GPU is that the repaired checkpoint is
        indistinguishable from what the paired non-compile run writes: same
        keys, same values. Anything peft finds in one it finds in the other.
        """
        from safetensors.torch import load_file

        from soup_cli.utils.peft_wiring import build_compile_prefix_callback

        repaired = _write_adapter(tmp_path / "checkpoint-100", prefix="_orig_mod.")
        _fire_on_save(build_compile_prefix_callback(), tmp_path, 100)
        reference = _write_adapter(tmp_path / "no-compile")

        got, want = load_file(str(repaired)), load_file(str(reference))
        assert set(got) == set(want)
        for key, tensor in want.items():
            assert torch.equal(got[key], tensor), key
        assert all(float(v.abs().max()) > 0 for k, v in got.items() if "lora_B" in k)

    def test_only_the_rank_that_wrote_the_file_rewrites(self, tmp_path):
        """``save_model`` writes the adapter only where ``args.should_save`` is
        true, but ``on_save`` is dispatched on EVERY rank. Unguarded, the 8 ranks
        of the run this was measured on would all rewrite one file at once."""
        from safetensors.torch import load_file

        from soup_cli.utils.peft_wiring import build_compile_prefix_callback

        path = _write_adapter(tmp_path / "checkpoint-100", prefix="_orig_mod.")
        before = set(load_file(str(path)))
        _fire_on_save(build_compile_prefix_callback(), tmp_path, 100, should_save=False)
        assert set(load_file(str(path))) == before

    def test_a_writing_rank_that_is_not_world_zero_still_repairs(self, tmp_path):
        """The reason the guard reads ``args.should_save`` and not
        ``state.is_world_process_zero``.

        ``TrainingArguments.should_save`` is ``local_process_index == 0`` when
        ``save_on_each_node`` is set and ``process_index == 0`` otherwise, and
        ``save_model`` gates the write on it. So under ``save_on_each_node`` the
        local rank 0 of every node after the first writes a checkpoint while
        reporting ``is_world_process_zero=False``, and a guard on world-zero
        would leave exactly those nodes holding an unrepaired adapter. Soup sets
        no such flag today, which is what makes this the kind of thing that goes
        unnoticed until someone does.
        """
        from safetensors.torch import load_file

        from soup_cli.utils.peft_wiring import build_compile_prefix_callback

        path = _write_adapter(tmp_path / "checkpoint-100", prefix="_orig_mod.")
        _fire_on_save(
            build_compile_prefix_callback(),
            tmp_path,
            100,
            should_save=True,
            world_zero=False,
        )

        keys = set(load_file(str(path)))
        assert not any(k.startswith("_orig_mod.") for k in keys), keys

    def test_a_checkpoint_without_an_adapter_is_not_an_error(self, tmp_path):
        """Full fine-tuning checkpoints carry no adapter file. Mid-run is a worse
        place to raise than the end of a run, so this must stay a no-op."""
        from soup_cli.utils.peft_wiring import build_compile_prefix_callback

        (tmp_path / "checkpoint-100").mkdir()
        _fire_on_save(build_compile_prefix_callback(), tmp_path, 100)

    def test_a_rewrite_that_fails_warns_and_keeps_training(
        self, tmp_path, monkeypatch, caplog
    ):
        """A broken rewrite must not kill a multi-hour run, but it must not pass
        in silence either, because a checkpoint left with the prefix is a dead
        adapter and that is the whole bug."""
        from soup_cli.utils import peft_wiring

        _write_adapter(tmp_path / "checkpoint-100", prefix="_orig_mod.")

        def boom(_output_dir):
            raise OSError("disk full")

        monkeypatch.setattr(peft_wiring, "strip_compile_prefix", boom)
        with caplog.at_level("WARNING"):
            _fire_on_save(peft_wiring.build_compile_prefix_callback(), tmp_path, 100)

        assert "checkpoint-100" in caplog.text, caplog.text

    def test_the_callback_answers_every_trainer_event(self):
        """#308. HF's ``CallbackHandler.call_event`` dispatches through
        ``getattr(cb, event)`` with no ``hasattr`` guard, so a duck-typed
        callback dies on the first ``on_epoch_begin`` rather than at wiring
        time. Inheriting ``TrainerCallback`` is what supplies the no-op stubs."""
        from transformers import TrainerCallback

        from soup_cli.utils.peft_wiring import build_compile_prefix_callback

        callback = build_compile_prefix_callback()
        assert isinstance(callback, TrainerCallback)
        callback.on_epoch_begin(
            types.SimpleNamespace(), types.SimpleNamespace(), types.SimpleNamespace()
        )

    def test_it_is_wired_only_when_compile_is_on(self):
        """CONTROL. Every ordinary run would otherwise carry a callback that
        reopens each checkpoint for nothing."""
        from soup_cli.utils.peft_wiring import attach_compile_prefix_callback

        class FakeTrainer:
            def __init__(self):
                self.callbacks = []

            def add_callback(self, callback):
                self.callbacks.append(callback)

        off = FakeTrainer()
        assert attach_compile_prefix_callback(
            off, types.SimpleNamespace(use_fsdp2_compile=False), "out"
        ) is False
        assert off.callbacks == []

        on = FakeTrainer()
        assert attach_compile_prefix_callback(
            on, types.SimpleNamespace(use_fsdp2_compile=True), "out"
        ) is True
        assert len(on.callbacks) == 1


# ==========================================================================
# #77 (low) — the --no-reexec hint dropped the user's own flags
# ==========================================================================
class TestNoReexecHintKeepsTheUsersFlags:
    """`--gpus 4 --fsdp full_shard --no-reexec` printed a command WITHOUT
    `--fsdp`, so following the hint literally trained without FSDP.

    The hint was hand-built as ``["soup", "train", "-c", config]`` a few lines
    above the auto-reexec path, which assembles the real flag list. Two copies of
    "what the user typed", and only one of them was maintained. The hint is now
    derived from the same ``script_args``, so they cannot drift.
    """

    def _advice(self, tmp_path, monkeypatch, extra_args):
        import re

        from typer.testing import CliRunner

        from soup_cli.cli import app
        from soup_cli.utils import topology as topo_mod

        monkeypatch.chdir(tmp_path)
        (tmp_path / "soup.yaml").write_text(
            "base: test/model\n"
            "task: sft\n"
            "data: {train: data.jsonl, format: alpaca}\n"
            "training: {epochs: 1, lr: 1e-4, batch_size: 1}\n",
            encoding="utf-8",
        )
        monkeypatch.setattr(
            topo_mod, "detect_topology",
            lambda: {"gpu_count": 4, "interconnect": "PCIe"},
        )
        monkeypatch.setattr(topo_mod, "resolve_num_gpus", lambda spec: 4)
        result = CliRunner().invoke(
            app,
            ["train", "--config", "soup.yaml", "--gpus", "4", "--no-reexec", "--yes",
             *extra_args],
        )
        # Rich wraps and may inject ANSI; strip both before matching, the way
        # every other CLI-output assertion in this repo has had to since v0.71.26.
        return re.sub(r"\x1b\[[0-9;]*m", "", result.output).replace("\n", " ")

    def test_the_hint_repeats_the_flags_the_user_passed(self, tmp_path, monkeypatch):
        out = self._advice(
            tmp_path, monkeypatch, ["--fsdp", "full_shard", "--trust-remote-code"]
        )
        if "Multi-GPU launch required" not in out:
            pytest.skip("did not reach the advisory (config/data validation ran first)")
        assert "--fsdp" in out, out
        assert "full_shard" in out, out
        assert "--trust-remote-code" in out, out

    def test_the_hint_does_not_tell_you_to_pass_no_reexec(self, tmp_path, monkeypatch):
        """Under `accelerate launch` the run is already distributed and never
        re-execs, so repeating the flag would be noise the user has to reason
        about. CONTROL for the derivation: it proves the hint is filtered rather
        than a raw echo of the reexec argv."""
        out = self._advice(tmp_path, monkeypatch, ["--fsdp", "full_shard"])
        if "Multi-GPU launch required" not in out:
            pytest.skip("did not reach the advisory (config/data validation ran first)")
        assert "--no-reexec" not in out, out


# ==========================================================================
# Liger was silently inert for every local model path
# ==========================================================================
class TestLigerDetectsArchitectureFromTheConfig:
    """`apply_liger_kernel` matched on a SUBSTRING OF THE MODEL PATH.

    It first tried `AutoLigerKernelForCausalLM._apply_liger_kernel(model_name)` —
    which **does not exist** in liger-kernel 0.8.1 (`AttributeError: type object
    ... has no attribute '_apply_liger_kernel'`, verified on the box) — and then
    fell back to `_apply_liger_manual(model_name.lower())`, which looks for
    "llama" / "mistral" / "qwen2" and friends **in the name**.

    So for any model loaded from a local directory the name carries no
    architecture, nothing matched, and the run printed "no matching architecture
    found" and trained without Liger. Silently, on a flag the user explicitly set.

    The architecture is in the config, where it cannot be renamed away: liger 0.8.1
    exposes a module-level `_apply_liger_kernel(model_type: str)`, and
    `AutoConfig.model_type` is exactly that string.
    """

    def _fake_liger(self, monkeypatch, calls):
        """Inject a stand-in liger so this runs where liger is not installed —
        which is the machine the maintainer develops on, and the reason the defect
        survived."""
        import sys
        import types

        mod = types.ModuleType("liger_kernel.transformers")

        def _apply(model_type, **kwargs):
            calls.append(model_type)

        mod._apply_liger_kernel = _apply
        mod._apply_liger_kernel_to_instance = lambda model, **kw: None
        parent = types.ModuleType("liger_kernel")
        parent.transformers = mod
        monkeypatch.setitem(sys.modules, "liger_kernel", parent)
        monkeypatch.setitem(sys.modules, "liger_kernel.transformers", mod)
        return calls

    def test_a_local_path_with_no_architecture_in_its_name_is_still_detected(
        self, tmp_path, monkeypatch
    ):
        from test_v07202 import _tiny_llama_dir

        from soup_cli.utils import liger as liger_mod

        weights, _, _ = _tiny_llama_dir(tmp_path)
        assert "llama" not in weights.lower(), (
            "the fixture path must NOT contain the architecture name, or this test "
            "passes for the substring matcher too and proves nothing"
        )
        calls: list = []
        self._fake_liger(monkeypatch, calls)
        monkeypatch.setattr(liger_mod, "check_liger_available", lambda: True)

        assert liger_mod.apply_liger_kernel(weights) is True
        assert calls == ["llama"], calls

    def test_an_unknown_architecture_still_reports_false(self, tmp_path, monkeypatch):
        """CONTROL. Detection by config must not turn into "always True" — an
        architecture liger does not support has to keep reporting False so the
        caller keeps printing its warning and does not set the TRL flag."""
        import json

        from soup_cli.utils import liger as liger_mod

        directory = tmp_path / "weird"
        directory.mkdir()
        (directory / "config.json").write_text(
            json.dumps({"model_type": "not_a_real_arch"}), encoding="utf-8"
        )
        calls: list = []
        self._fake_liger(monkeypatch, calls)

        def _raise(model_type, **kwargs):
            raise NotImplementedError(model_type)

        import sys

        sys.modules["liger_kernel.transformers"]._apply_liger_kernel = _raise
        monkeypatch.setattr(liger_mod, "check_liger_available", lambda: True)

        assert liger_mod.apply_liger_kernel(str(directory)) is False


# ==========================================================================
# #351 review: the normalisation has to reach the file before anything
# that PUBLISHES it does
# ==========================================================================
class TestTheNormalisationRunsBeforeAnythingThatPublishes:
    """``HFPushCallback.on_save`` uploads ``checkpoint-{step}`` on the same event
    this normalisation runs on, so the order the two are dispatched in decides
    what ``--push-as`` puts on the Hub.

    ``CallbackHandler.add_callback`` appends and ``call_event`` iterates in
    insertion order, and ``--push-as`` adds its callback to the trainer after
    ``setup()`` has returned (``commands/train.py``). Attaching the
    normalisation in ``setup()`` is therefore what keeps it in front: attached in
    ``train()`` instead, it landed second, the Hub received the prefixed adapter
    and only local disk got the repaired one. That is the same dead-adapter
    defect on the path where it is least recoverable, since what is published is
    the dead file.

    Asserted through the real ``callback_handler`` rather than by comparing list
    indices, because the property that matters is what a later callback SEES.
    """

    def test_a_later_added_on_save_sees_the_repaired_keys(self, tmp_path, monkeypatch):
        from pathlib import Path

        from safetensors.torch import load_file
        from transformers import TrainerCallback

        wrapper = _sft_wrapper(tmp_path, monkeypatch, use_fsdp2_compile=True)
        trainer = wrapper.trainer
        seen = {}

        class Spy(TrainerCallback):
            """Stands in for ``HFPushCallback``: reads the checkpoint it is told
            was just written, which is what an upload does."""

            def on_save(self, args, state, control, **kwargs):
                adapter = (
                    Path(args.output_dir)
                    / f"checkpoint-{state.global_step}"
                    / "adapter_model.safetensors"
                )
                seen["keys"] = set(load_file(str(adapter)))

        # Exactly what `--push-as` does, and it can only ever happen after
        # `setup()` has run.
        trainer.add_callback(Spy())

        _write_adapter(
            Path(trainer.args.output_dir) / "checkpoint-100", prefix="_orig_mod."
        )
        trainer.state.global_step = 100
        trainer.control = trainer.callback_handler.on_save(
            trainer.args, trainer.state, trainer.control
        )

        assert seen, "the spy's on_save never fired; the wiring changed shape"
        assert not any(k.startswith("_orig_mod.") for k in seen["keys"]), (
            f"a callback added after setup() saw the PREFIXED adapter, so "
            f"--push-as would publish a dead one: {sorted(seen['keys'])}"
        )

    def test_the_callback_is_attached_by_setup_not_by_train(self, tmp_path, monkeypatch):
        """The position itself, so a move back into ``train()`` fails here rather
        than only in the subtler assertion above."""
        wrapper = _sft_wrapper(tmp_path, monkeypatch, use_fsdp2_compile=True)
        names = [
            type(cb).__name__ for cb in wrapper.trainer.callback_handler.callbacks
        ]
        assert "CompilePrefixCallback" in names, names