Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.
Python cliconsumer-gpudpofine-tuningggufhuggingfacellmllmopslocal-ailocal-llmloralow-vrammachine-learningollamapeftpythonpytorchqlorasfttransformers
feat(mlx): native SimPO on Apple Silicon (#1999) (#2097)
dryn85-maxcd3ad796b497
diff --git a/changelog.d/0.76.0/2097.added.md b/changelog.d/0.76.0/2097.added.mdnew file mode 100644index 00000000..30602ab1--- /dev/null+++ b/changelog.d/0.76.0/2097.added.md@@ -0,0 +1,12 @@+- **SimPO runs natively on Apple Silicon: `task: simpo` with `backend: mlx` (#1999 by @dryn85-max+ in #2097).** It was refused at config load. Soup's own MLX loop now trains TRL 0.29.1's CPO+ `simpo` objective: `-logsigmoid(0.1 * (a - b) - simpo_gamma)` on mean completion+ log-probabilities, with beta fixed at `CPOConfig`'s default 0.1 as on the transformers backend,+ plus `cpo_alpha` times the chosen-completion NLL. No reference model is needed. Rows are+ tokenized as TRL's CPO trainer does, so rows that fit in `max_length` train on the same tokens on+ either backend; ties train as in TRL. The adapter loads with `mlx_lm.load`. `--resume` refuses+ a checkpoint from another task or base. A nonfinite loss stops the run. `quantization: 8bit`,+ `lora.r: 0`, non-text modalities and LoRA+ are refused on MLX, and `task: preference` stays+ refused there, now naming `task: simpo` / `task: orpo` / `task: dpo`. Try:+ `soup train --config soup.yaml` with `task: simpo`, `backend: mlx` and+ `base: mlx-community/Qwen2.5-0.5B-Instruct-4bit`.diff --git a/docs/backends-and-ops.md b/docs/backends-and-ops.mdindex 151d640c..a5e9f61e 100644--- a/docs/backends-and-ops.md+++ b/docs/backends-and-ops.md@@ -136,7 +136,7 @@ training:alpha: 32```-MLX backend supports SFT, DPO and ORPO. Any other task with `backend: mlx` (`grpo`, `ipo`, `kto`, `simpo`, `preference`, ...) is refused when the config is loaded, with an error naming the task. Requires `mlx-lm >= 0.31.3`. Use `soup recipes search mlx` for ready-made Apple Silicon configs.+MLX backend supports SFT, DPO, ORPO and SimPO. Any other task with `backend: mlx` (`grpo`, `ipo`, `kto`, `preference`, ...) is refused when the config is loaded, with an error naming the task. Requires `mlx-lm >= 0.31.3`. Use `soup recipes search mlx` for ready-made Apple Silicon configs.#### DPO on MLX@@ -239,6 +239,62 @@ training:`backend: mlx` (use `task: orpo`). The other settings MLX DPO does not applyare not applied here either.+#### SimPO on MLX++`task: simpo` with `backend: mlx` runs the same loop as MLX DPO with SimPO's+objective, matched to TRL 0.29.1's CPO trainer with `loss_type: simpo` (the one+the transformers backend runs). With `a` and `b` the mean log-probability of the+chosen and rejected completion tokens:++`loss = mean(-logsigmoid(beta * (a - b) - simpo_gamma)) + cpo_alpha * NLL`++Qualified on Qwen2.5-0.5B-Instruct in bf16 and 4-bit on an M4 Pro: one epoch on+600 UltraFeedback pairs took about 4.5 minutes and 7.5-8.6 GB, and the mean+log-probability margin on 200 held-out pairs rose 1.9x (bf16) and 3.1x (4-bit),+while held-out pairwise accuracy barely moved (0.540 to 0.565, 0.545 to 0.550).+That shows the objective moving, not alignment quality.++- **Beta is 0.1, fixed.** Soup's transformers backend passes no beta to TRL, so+ `CPOConfig`'s default 0.1 applies; MLX uses the same value and records it as+ `simpo_beta` in the adapter metadata. `dpo_beta` and `orpo_beta` are not+ read. The margin `simpo_gamma` sits outside the temperature, as in TRL (which+ writes it `beta * (a - b - simpo_gamma / beta)`). Label smoothing is TRL's+ default 0.+- **The NLL covers the chosen completion only.** CPO's labels mask the prompt,+ so the NLL is a token mean over the chosen completion tokens of the whole+ batch (not a mean of per-row means). This differs from ORPO, whose NLL+ includes the prompt. `cpo_alpha` weights it (`> 0`, as on the transformers+ backend). With `gradient_accumulation_steps > 1` the NLL is a token mean per+ micro-batch and the micro-batches are averaged with equal weight, as the+ transformers Trainer does for TRL's CPO.+- **Tokens** are ORPO's above: TRL's CPO and ORPO trainers tokenize rows with+ the same code, so rows that fit in `data.max_length` train on the same tokens+ on either backend. Over-long rows are cut with Soup's cap (prompt to+ `max_length // 2`, keeping its end, answers to the remaining room). The+ transformers SimPO path applies no prompt cap: TRL 0.29.1 cuts only the+ answers there, and Soup refuses a row whose shorter answer that cut empties.+- **Ties train**, as in TRL: the preference term is the constant+ `softplus(simpo_gamma)` and the NLL still learns.+- **Stops on a nonfinite loss** instead of training on NaN.+- **LoRA dropout applies**, as in MLX DPO above; set `lora.dropout: 0` to+ match TRL's `disable_dropout`.+- **Reported:** loss, NLL, chosen and rejected rewards (`0.1 * logp`, as TRL+ logs them), reward accuracy and margin, and the mean chosen and rejected+ log-probabilities, per logged step and in the run result (`val_*` for the+ validation split). TRL's mean-logit metrics are not computed: the vocabulary+ head runs in chunks and the full logits are never held.+- **Saved:** `adapters.safetensors` and `adapter_config.json` (with `task:+ simpo`, `simpo_beta`, `simpo_gamma`, `cpo_alpha` and the base model),+ loadable with `mlx_lm.load(base, adapter_path=output)`, written as MLX ORPO+ writes them. `--resume` refuses a checkpoint without that metadata, from+ another task (ORPO and DPO included), or from another base, before any tensor+ loads.+- **Refused at load:** `quantization: 8bit`, `training.lora.r: 0`, non-text+ `modality`, LoRA+, `data.chat_template` (as for MLX DPO and ORPO), and+ `task: preference` with `preference_loss: simpo` on+ `backend: mlx` (use `task: simpo`). The other settings MLX DPO does not apply+ are not applied here either.+#### DoRA on MLXFor text SFT, set `training.lora.use_dora: true` to use MLX-LM's native@@ -300,7 +356,7 @@ what ran is recoverable from the output directory.### Transformers on MPS-The regular `backend: transformers` path can run more than MLX's SFT, DPO and ORPO+The regular `backend: transformers` path can run more than MLX's SFT, DPO, ORPO and SimPOsurface. On a live MPS runtime that accepts bfloat16, Soup enables BF16 autocastfor the hardware-validated text trainers: SFT, DPO, GRPO/RLVR, reward modelling,and PRM.diff --git a/docs/models.md b/docs/models.mdindex 5f473d1d..048b036b 100644--- a/docs/models.md+++ b/docs/models.md@@ -144,7 +144,7 @@ spelling is unchanged. See [the README's install section](../README.md#1-install| `fast` | `pip install "soup-cli[fast]"` | Unsloth backend (2-5x faster, lower VRAM) || `vision` | `pip install "soup-cli[vision]"` | Vision / multimodal fine-tuning (Pillow) || `audio` | `pip install "soup-cli[audio]"` | Audio / speech fine-tuning (librosa, soundfile) |-| `mlx` | `pip install "soup-cli[mlx]"` | Standalone Apple Silicon SFT, DPO and ORPO backend for local data; `[train]` is not required |+| `mlx` | `pip install "soup-cli[mlx]"` | Standalone Apple Silicon SFT, DPO, ORPO and SimPO backend for local data; `[train]` is not required || `qat` | `pip install "soup-cli[qat]"` | torchao: FP8 training (`quantization_aware: fp8`), FP8 attention, NVFP4, the `ao_adamw_*` optimizers and `soup export --format torchao` || `serve` | `pip install "soup-cli[serve]"` | Inference server (FastAPI + uvicorn) || `serve-fast` | `pip install "soup-cli[serve-fast]"` | vLLM inference backend (2-4x throughput) |diff --git a/docs/training.md b/docs/training.mdindex 7269910d..18378fa6 100644--- a/docs/training.md+++ b/docs/training.md@@ -1857,6 +1857,9 @@ as `0.0`. `soup train` stops before the first step, naming how many rows andwhich, with the `data.max_length` involved. Raise `data.max_length`, balance thepair, or drop the row.+On Apple Silicon, `backend: mlx` runs SimPO natively with the same objective; see+[SimPO on MLX](backends-and-ops.md#simpo-on-mlx).+## IPO Training (Regularized Preference)diff --git a/src/soup_cli/config/backend_support.py b/src/soup_cli/config/backend_support.pyindex 954391dc..18bc8b7f 100644--- a/src/soup_cli/config/backend_support.py+++ b/src/soup_cli/config/backend_support.py@@ -19,10 +19,10 @@ drift -- every entry must name a real field, and every dotted field name inthe truth; it can only stop it rotting, which is the failure that produced #749.Scope today: ``task=sft`` on ``backend=mlx`` and on ``backend=unsloth`` (#1124;-both apply to every modality), ``task=dpo`` and ``task=orpo`` on ``backend=mlx``, plus-multimodal vision and audio on ``backend=transformers`` (#1156). Other combinations report-nothing rather than guessing, which is the honest default for a table that has not been-reviewed for them.+both apply to every modality), ``task=dpo``, ``task=orpo`` and ``task=simpo`` on+``backend=mlx``, plus multimodal vision and audio on ``backend=transformers`` (#1156).+Other combinations report nothing rather than guessing, which is the honest default for+a table that has not been reviewed for them."""from __future__ import annotations@@ -231,12 +231,15 @@ _MLX_DPO: tuple[SupportEntry, ...] = tuple(#: MLX ORPO runs MLX DPO's loop with its own loss, so it ignores the same fields._MLX_ORPO: tuple[SupportEntry, ...] = _MLX_DPO+#: MLX SimPO is MLX ORPO's tokenization and MLX DPO's loop: the same fields.+_MLX_SIMPO: tuple[SupportEntry, ...] = _MLX_ORPO#: ``(task, backend, modality)`` -> the settings that combination does not honour.REGISTRY: dict[tuple[str, str, str], tuple[SupportEntry, ...]] = {("sft", "mlx", "text"): _MLX_SFT,("dpo", "mlx", "text"): _MLX_DPO,("orpo", "mlx", "text"): _MLX_ORPO,+ ("simpo", "mlx", "text"): _MLX_SIMPO,("sft", "unsloth", "text"): _UNSLOTH_SFT,("sft", "transformers", "vision"): _TRANSFORMERS_VISION_SFT,("sft", "transformers", "audio"): _TRANSFORMERS_AUDIO_SFT,@@ -273,6 +276,15 @@ TRAINER_MODULES: dict[tuple[str, str, str], tuple[str, ...]] = {"soup_cli/trainer/_trl_compat.py","soup_cli/trainer/loss_summary.py",),+ ("simpo", "mlx", "text"): (+ "soup_cli/trainer/mlx_simpo.py",+ "soup_cli/trainer/mlx_orpo.py",+ "soup_cli/trainer/mlx_dpo.py",+ "soup_cli/trainer/mlx_sft.py",+ "soup_cli/trainer/mlx_optim.py",+ "soup_cli/trainer/_trl_compat.py",+ "soup_cli/trainer/loss_summary.py",+ ),("orpo", "mlx", "text"): ("soup_cli/trainer/mlx_orpo.py","soup_cli/trainer/mlx_dpo.py",diff --git a/src/soup_cli/config/schema.py b/src/soup_cli/config/schema.pyindex 5a46507b..16e5e4a3 100644--- a/src/soup_cli/config/schema.py+++ b/src/soup_cli/config/schema.py@@ -4941,9 +4941,9 @@ UNSLOTH_SETUP_TASKS: frozenset[str] = frozenset({})#: The tasks with a trainer on ``backend: mlx``: SFT through mlx-lm's tuner and-#: DPO through Soup's own loop (``trainer/mlx_dpo.py``). Validators that defer+#: DPO, ORPO and SimPO through Soup's own loop (``trainer/mlx_dpo.py``). Validators that defer#: to :meth:`SoupConfig._validate_mlx_task_support` read this, not ``"sft"``.-MLX_TASKS: frozenset[str] = frozenset({"sft", "dpo", "orpo"})+MLX_TASKS: frozenset[str] = frozenset({"sft", "dpo", "orpo", "simpo"})#: #1497 — the tasks whose trainer reads ``training.freeze_layers`` /#: ``training.freeze_ratio``: ``SFTTrainerWrapper._setup_transformers``, which@@ -8290,22 +8290,23 @@ class SoupConfig(BaseModel):@model_validator(mode="after")def _validate_mlx_task_support(self) -> "SoupConfig":- """MLX backend supports sft, dpo and orpo.+ """MLX backend supports sft, dpo, orpo and simpo.- SFT runs through mlx-lm's tuner; DPO and ORPO through Soup's own loop- in ``trainer/mlx_dpo.py`` / ``trainer/mlx_orpo.py``. Every other task --- GRPO included, whose MLX wrapper is still scaffolding -- is refused here- rather than at ``train()`` time.+ SFT runs through mlx-lm's tuner; DPO, ORPO and SimPO through Soup's own+ loop in ``trainer/mlx_dpo.py`` / ``trainer/mlx_orpo.py`` /+ ``trainer/mlx_simpo.py``. Every other task -- GRPO included, whose MLX+ wrapper is still scaffolding -- is refused here rather than at+ ``train()`` time."""if self.backend != "mlx":return self- if self.task in ("dpo", "orpo"):+ if self.task in ("dpo", "orpo", "simpo"):# The native preference loops tokenize text and chat rows only and# train a LoRA adapter. DPO's reference log-probs come from the# policy before training, which is the base model only because a- # LoRA adapter's B starts at zero; ORPO trains LoRA only (no full- # fine-tune on MLX yet).- name = self.task.upper()+ # LoRA adapter's B starts at zero; ORPO and SimPO train LoRA only+ # (no full fine-tune on MLX yet).+ name = {"dpo": "DPO", "orpo": "ORPO", "simpo": "SimPO"}[self.task]if self.modality != "text":raise ValueError(f"modality='{self.modality}' is not supported by task='{self.task}' "@@ -8317,26 +8318,30 @@ class SoupConfig(BaseModel):"the reference model is the untrained LoRA policy, which needs ""a LoRA adapter to exist"if self.task == "dpo"- else "MLX ORPO trains a LoRA adapter; full fine-tuning is not "+ else f"MLX {name} trains a LoRA adapter; full fine-tuning is not ""implemented on backend='mlx'")raise ValueError(f"training.lora.r must be at least 1 for task='{self.task}' on "f"backend='mlx': {why}.")- if self.task == "orpo" and self.training.quantization == "8bit":+ if self.task in ("orpo", "simpo") and self.training.quantization == "8bit":raise ValueError(- "training.quantization='8bit' is not supported by task='orpo' on "- "backend='mlx': use quantization: none, or a pre-quantized "+ f"training.quantization='8bit' is not supported by task='{self.task}' "+ "on backend='mlx': use quantization: none, or a pre-quantized ""mlx-community 4-bit base with quantization: 4bit.")return selfif self.task in MLX_TASKS:return self+ hint = ""+ native = {"dpo", "orpo", "simpo"}+ if self.task == "preference" and self.training.preference_loss in native:+ hint = f" Use task: {self.training.preference_loss} for the native MLX trainer."raise ValueError(- f"MLX backend only ships SFT, DPO and ORPO; task='{self.task}' has no "- "trainer on backend='mlx'. "- f"Use backend=transformers for task={self.task}."+ f"MLX backend only ships SFT, DPO, ORPO and SimPO; task='{self.task}' has no "+ "trainer on backend='mlx'."+ + (hint or f" Use backend=transformers for task={self.task}."))@model_validator(mode="after")diff --git a/src/soup_cli/trainer/mlx_dpo.py b/src/soup_cli/trainer/mlx_dpo.pyindex 0061cb3d..60a811af 100644--- a/src/soup_cli/trainer/mlx_dpo.py+++ b/src/soup_cli/trainer/mlx_dpo.py@@ -483,9 +483,10 @@ class MLXDPOTrainerWrapper(MLXSFTTrainerWrapper):# -- objective hooks --------------------------------------------------## The loop below is shared by every native preference objective on MLX- # (DPO here, ORPO in mlx_orpo.py). A subclass changes the tokenized- # example, the arrays a batch needs, what is prepared before the first- # update, the loss and its metrics -- nothing else.+ # (DPO here, ORPO in mlx_orpo.py, SimPO in mlx_simpo.py). A subclass+ # changes the tokenized example, the arrays a batch needs, what is+ # prepared before the first update, the loss and its metrics -- nothing+ # else.#: Name used in messages and the adapter metadata.objective = "DPO"diff --git a/src/soup_cli/trainer/mlx_orpo.py b/src/soup_cli/trainer/mlx_orpo.pyindex 87263d27..9e77e0b9 100644--- a/src/soup_cli/trainer/mlx_orpo.py+++ b/src/soup_cli/trainer/mlx_orpo.py@@ -356,37 +356,40 @@ class MLXORPOTrainerWrapper(MLXDPOTrainerWrapper):}def _load_checkpoint_weights(self, checkpoint_path: str) -> None:- """Warm start only from an ORPO adapter of the same base.-- The checkpoint's directory must hold the ``adapter_config.json`` an MLX- ORPO run writes (it is written before the first step, so step- checkpoints of a run that stopped early have it too). A DPO or SFT- adapter, a different base model, or no metadata at all is refused- before any tensor loads; the tensor names and shapes are then checked- by MLX SFT's loader.+ """Warm start only from an adapter of this objective and base.++ The checkpoint's directory must hold the ``adapter_config.json`` the+ run writes (it is written before the first step, so step checkpoints of+ a run that stopped early have it too). An adapter of another task+ (another preference objective, or SFT), a different base model, or no+ metadata at all is refused before any tensor loads; the tensor names+ and shapes are then checked by MLX SFT's loader. Subclasses reuse it+ through ``objective`` and their metadata's ``task``."""import jsonfrom pathlib import Path+ name = self.objective+ task = self._objective_metadata()["task"]meta_path = Path(checkpoint_path).resolve().parent / "adapter_config.json"if not meta_path.is_file():raise ValueError(- f"MLX ORPO: no adapter_config.json beside {checkpoint_path}; cannot "- "confirm it is an ORPO adapter of this base model"+ f"MLX {name}: no adapter_config.json beside {checkpoint_path}; cannot "+ f"confirm it is a task={task!r} adapter of this base model")try:meta = json.loads(meta_path.read_text(encoding="utf-8"))except (OSError, ValueError) as exc:- raise ValueError(f"MLX ORPO: unreadable {meta_path}: {exc}") from exc- if meta.get("task") != "orpo":+ raise ValueError(f"MLX {name}: unreadable {meta_path}: {exc}") from exc+ if meta.get("task") != task:raise ValueError(- f"MLX ORPO: {checkpoint_path} was trained with task="- f"{meta.get('task')!r}, not 'orpo'; start a fresh run or resume "+ f"MLX {name}: {checkpoint_path} was trained with task="+ f"{meta.get('task')!r}, not {task!r}; start a fresh run or resume ""with the matching task")if str(meta.get("model")) != str(self.config.base):raise ValueError(- f"MLX ORPO: {checkpoint_path} belongs to base {meta.get('model')!r}, "+ f"MLX {name}: {checkpoint_path} belongs to base {meta.get('model')!r}, "f"not {self.config.base!r}")super()._load_checkpoint_weights(checkpoint_path)diff --git a/src/soup_cli/trainer/mlx_routing.py b/src/soup_cli/trainer/mlx_routing.pyindex fdb60a16..59696961 100644--- a/src/soup_cli/trainer/mlx_routing.py+++ b/src/soup_cli/trainer/mlx_routing.py@@ -24,6 +24,10 @@ class _MLXRegistry(dict):from soup_cli.trainer.mlx_orpo import MLXORPOTrainerWrapperreturn MLXORPOTrainerWrapper+ if task == "simpo":+ from soup_cli.trainer.mlx_simpo import MLXSimPOTrainerWrapper++ return MLXSimPOTrainerWrapperif task == "grpo":from soup_cli.trainer.mlx_grpo import MLXGRPOTrainerWrapper@@ -31,13 +35,13 @@ class _MLXRegistry(dict):raise KeyError(task)def __contains__(self, key: object) -> bool:- return key in ("sft", "dpo", "orpo", "grpo")+ return key in ("sft", "dpo", "orpo", "simpo", "grpo")def __getitem__(self, key: str):return self._resolve(key)def keys(self): # noqa: D401- return ("sft", "dpo", "orpo", "grpo")+ return ("sft", "dpo", "orpo", "simpo", "grpo")MLX_TRAINER_REGISTRY: _MLXRegistry = _MLXRegistry()diff --git a/src/soup_cli/trainer/mlx_simpo.py b/src/soup_cli/trainer/mlx_simpo.pynew file mode 100644index 00000000..70dc3675--- /dev/null+++ b/src/soup_cli/trainer/mlx_simpo.py@@ -0,0 +1,130 @@+"""MLX SimPO trainer — reference-free, length-normalised preference optimization on Apple Silicon.++The objective is TRL 0.29.1's ``experimental/cpo/cpo_trainer.py`` with+``loss_type="simpo"``, the trainer Soup's transformers backend runs for+``task: simpo`` (``trainer/simpo.py`` forwards ``simpo_gamma`` and+``cpo_alpha`` and no beta):++- ``a``, ``b``: the **mean** log-probability of the chosen / rejected completion+ tokens (``get_batch_logps(..., average_log_prob=True)``);+- preference term ``-logsigmoid(beta * (a - b) - simpo_gamma)`` per pair, the+ margin outside the temperature (TRL writes it ``beta * (a - b - gamma / beta)``);+ ``label_smoothing`` is ``CPOConfig``'s default 0, so its second term vanishes;+- ``beta`` is ``CPOConfig.beta``'s default, :data:`SIMPO_BETA`; Soup exposes no+ knob for it (``dpo_beta`` / ``orpo_beta`` are other objectives');+- loss = ``mean(preference) + cpo_alpha * NLL``, where the NLL is a token-mean+ cross-entropy over the chosen **completion** tokens only (CPO's labels mask+ the prompt with ``-100``), across the batch.++Tokenization is ORPO's: TRL's CPO and ORPO ``tokenize_row`` /+``build_tokenized_answer`` are the same code, so the rows, the masks and Soup's+length cap come from :mod:`soup_cli.trainer.mlx_orpo`. The completion mask is+both the score mask and the NLL mask. The loop is MLX DPO's.+"""++from __future__ import annotations++from typing import Any++from soup_cli.trainer.mlx_orpo import (+ ORPO_TRUNCATION_MODE,+ MLXORPOTrainerWrapper,+ collate_orpo,+ orpo_terms,+)++#: ``CPOConfig.beta``'s default in TRL 0.29.1, the SimPO temperature Soup's+#: transformers backend trains with.+SIMPO_BETA = 0.1+++def simpo_terms(model: Any, tokens: Any, completion: Any, *,+ head_split: Any = None) -> tuple[Any, Any, Any]:+ """``(chosen mean logps, rejected mean logps, chosen NLL)`` for a collated batch.++ ORPO's terms with the completion mask in place of the sequence mask: CPO's+ NLL reads the same labels as the scores, so the prompt is not in it.+ """+ return orpo_terms(model, tokens, completion, completion, head_split=head_split)+++def simpo_loss(+ chosen_mean_logps: Any,+ rejected_mean_logps: Any,+ chosen_nll: Any,+ *,+ beta: float,+ gamma: float,+ cpo_alpha: float,+) -> tuple[Any, dict]:+ """TRL's SimPO loss and its metrics from per-pair mean completion log-probs.++ Returns ``(loss, metrics)`` with ``preference_loss`` (the mean preference+ term, which TRL does not log) and TRL's logged quantities: ``nll_loss``,+ ``reward_chosen`` / ``reward_rejected`` (``beta * logp``),+ ``reward_accuracy`` (strict ``>``), ``reward_margin``, and+ ``logps_chosen`` / ``logps_rejected``, all batch means.+ """+ import mlx.core as mx+ import mlx.nn as nn++ a = chosen_mean_logps.astype(mx.float32)+ b = rejected_mean_logps.astype(mx.float32)+ logits = (a - b) - gamma / beta+ pref = -nn.log_sigmoid(beta * logits)+ loss = pref.mean() + cpo_alpha * chosen_nll+ chosen_rewards, rejected_rewards = beta * a, beta * b+ metrics = {+ "preference_loss": pref.mean(),+ "nll_loss": chosen_nll,+ "reward_chosen": chosen_rewards.mean(),+ "reward_rejected": rejected_rewards.mean(),+ "reward_accuracy": (chosen_rewards > rejected_rewards).astype(mx.float32).mean(),+ "reward_margin": (chosen_rewards - rejected_rewards).mean(),+ "logps_chosen": a.mean(),+ "logps_rejected": b.mean(),+ }+ return loss, metrics+++class MLXSimPOTrainerWrapper(MLXORPOTrainerWrapper):+ """SimPO on MLX: MLX ORPO's tokens, MLX DPO's loop, CPO's simpo loss."""++ objective = "SimPO"+ metric_names = (+ "preference_loss", "nll_loss", "reward_chosen", "reward_rejected", "reward_accuracy",+ "reward_margin", "logps_chosen", "logps_rejected",+ )++ def _batch_arrays(self, examples: list) -> tuple:+ import mlx.core as mx++ tokens, completion, _ = collate_orpo(examples)+ return mx.array(tokens), mx.array(completion)++ def _loss(self, model: Any, arrays: tuple, split: str, rows: Any) -> tuple:+ tokens, completion = arrays+ a, b, nll = simpo_terms(model, tokens, completion, head_split=self._head_split)+ tcfg = self.config.training+ return simpo_loss(+ a, b, nll, beta=SIMPO_BETA, gamma=float(tcfg.simpo_gamma),+ cpo_alpha=float(tcfg.cpo_alpha),+ )++ def _format_metrics(self, metrics: dict) -> str:+ return (+ f"nll {metrics['nll_loss']:.4f}, reward acc {metrics['reward_accuracy']:.2f}, "+ f"margin {metrics['reward_margin']:.4f}"+ )++ def _objective_metadata(self) -> dict:+ tcfg = self.config.training+ return {+ "task": "simpo",+ "simpo_beta": SIMPO_BETA,+ "simpo_gamma": float(tcfg.simpo_gamma),+ "cpo_alpha": float(tcfg.cpo_alpha),+ "simpo_loss": "trl-0.29.1 cpo simpo + cpo_alpha * chosen completion NLL",+ "reference": "none (reference-free)",+ "truncation_mode": ORPO_TRUNCATION_MODE,+ }diff --git a/tests/test_issue1213_dispatch_parity.py b/tests/test_issue1213_dispatch_parity.pyindex 9a88a9fe..77f16f39 100644--- a/tests/test_issue1213_dispatch_parity.py+++ b/tests/test_issue1213_dispatch_parity.py@@ -77,12 +77,13 @@ _EXPECTED = {"asr": "AsrTrainerWrapper",}-#: MLX rows resolve through ``mlx_routing``. Same four tasks the registry+#: MLX rows resolve through ``mlx_routing``. Same five tasks the registry#: accepts; the name is what ``resolve_trainer`` would have imported._MLX_EXPECTED = {"sft": "MLXSFTTrainerWrapper","dpo": "MLXDPOTrainerWrapper","orpo": "MLXORPOTrainerWrapper",+ "simpo": "MLXSimPOTrainerWrapper","grpo": "MLXGRPOTrainerWrapper",}diff --git a/tests/test_issue1358_vocab_expansion_refused.py b/tests/test_issue1358_vocab_expansion_refused.pyindex 4fa65be1..1ab20e12 100644--- a/tests/test_issue1358_vocab_expansion_refused.py+++ b/tests/test_issue1358_vocab_expansion_refused.py@@ -135,10 +135,10 @@ class TestTheOtherPathsThatDropThem:assert "data.add_new_tokens is not applied on backend='mlx'" in message, message@pytest.mark.parametrize("field", _FIELDS)- @pytest.mark.parametrize("task", ["dpo", "orpo"])+ @pytest.mark.parametrize("task", ["dpo", "orpo", "simpo"])def test_mlx_preference_loops(self, task, field):- """The native DPO and ORPO loops tokenize with the base tokenizer as it is;- the refusal rests on both being in ``MLX_TASKS`` (review of #2073)."""+ """The native DPO, ORPO and SimPO loops tokenize with the base tokenizer as+ it is; the refusal rests on each being in ``MLX_TASKS`` (review of #2073)."""message = _refusal(_yaml(task, backend="mlx", field=field, data={"format": "dpo"}))assert f"data.{field} is not applied on backend='mlx'" in message, messagediff --git a/tests/test_mlx_chat_template_refusal.py b/tests/test_mlx_chat_template_refusal.pyindex 50efde8a..5ac55198 100644--- a/tests/test_mlx_chat_template_refusal.py+++ b/tests/test_mlx_chat_template_refusal.py@@ -19,7 +19,7 @@ TEMPLATES = ["chatml", "llama3", "{% for m in messages %}{{ m['content'] }}{% enclass TestMlxChatTemplateRefusal:- @pytest.mark.parametrize("task", ["sft", "dpo", "orpo"])+ @pytest.mark.parametrize("task", ["sft", "dpo", "orpo", "simpo"])@pytest.mark.parametrize("template", TEMPLATES)def test_mlx_with_chat_template_refused_at_config_load(self, template: str, task: str) -> None:with pytest.raises(ValueError) as exc_info:@@ -35,7 +35,7 @@ class TestMlxChatTemplateRefusal:assert "backend: transformers" in messageassert "Remove data.chat_template" in message- @pytest.mark.parametrize("task", ["sft", "dpo", "orpo"])+ @pytest.mark.parametrize("task", ["sft", "dpo", "orpo", "simpo"])def test_empty_chat_template_still_loads_on_mlx(self, task: str) -> None:# The field validator normalises "" to None, which is not an override.config = SoupConfig(@@ -47,7 +47,7 @@ class TestMlxChatTemplateRefusal:)assert config.data.chat_template is None- @pytest.mark.parametrize("task", ["sft", "dpo", "orpo"])+ @pytest.mark.parametrize("task", ["sft", "dpo", "orpo", "simpo"])@pytest.mark.parametrize("template", TEMPLATES)def test_transformers_backend_still_accepts_chat_template(self, template: str, task: str,@@ -62,7 +62,7 @@ class TestMlxChatTemplateRefusal:)assert config.data.chat_template == template- @pytest.mark.parametrize("task", ["sft", "dpo", "orpo"])+ @pytest.mark.parametrize("task", ["sft", "dpo", "orpo", "simpo"])def test_mlx_without_chat_template_still_allowed(self, task: str) -> None:config = SoupConfig(base="test/model",diff --git a/tests/test_mlx_dpo.py b/tests/test_mlx_dpo.pyindex db3cb98c..26d43d84 100644--- a/tests/test_mlx_dpo.py+++ b/tests/test_mlx_dpo.py@@ -468,8 +468,9 @@ class TestConfig:fields = {entry.field for entry in check_config(cfg)}assert fields == {"training.use_galore"}- # orpo has its own MLX trainer since #2004 (tests/test_mlx_orpo.py).- @pytest.mark.parametrize("task", ["ipo", "kto", "simpo", "ppo"])+ # orpo and simpo have their own MLX trainers since #2004 / #1999+ # (tests/test_mlx_orpo.py, tests/test_mlx_simpo.py).+ @pytest.mark.parametrize("task", ["ipo", "kto", "ppo"])def test_other_preference_tasks_still_refused(self, task):from soup_cli.config.loader import load_config_from_string@@ -477,6 +478,15 @@ class TestConfig:with pytest.raises(Exception, match="MLX backend"):load_config_from_string(yaml_str)+ def test_the_preference_alias_is_refused_naming_task_dpo(self):+ from soup_cli.config.loader import load_config_from_string++ alias = (_MLX_DPO_YAML.replace("task: dpo", "task: preference")+ .replace(" lr: 1e-6\n", " lr: 1e-6\n preference_loss: dpo\n"))+ with pytest.raises(Exception, match="task='preference' has no trainer") as info:+ load_config_from_string(alias)+ assert "Use task: dpo for the native MLX trainer." in str(info.value)+# --------------------------------------------------------------------------# End-to-end on a tiny random model (needs mlx + mlx_lm)diff --git a/tests/test_mlx_orpo.py b/tests/test_mlx_orpo.pyindex 4c25bc9a..4d82e670 100644--- a/tests/test_mlx_orpo.py+++ b/tests/test_mlx_orpo.py@@ -631,6 +631,7 @@ class TestResume:@pytest.mark.parametrize("change, match", [({"task": "dpo"}, "task='dpo'"),+ ({"task": "simpo"}, "MLX ORPO: .* task='simpo', not 'orpo'"),({"model": "other-base"}, "belongs to base"),(None, "no adapter_config.json"),])@@ -736,8 +737,9 @@ class TestConfig:alias = (_ORPO_YAML.replace("task: orpo", "task: preference").replace(" orpo_beta: 0.1\n", " preference_loss: orpo\n"))- with pytest.raises(Exception, match="task='preference' has no trainer"):+ with pytest.raises(Exception, match="task='preference' has no trainer") as info:load_config_from_string(alias)+ assert "Use task: orpo for the native MLX trainer." in str(info.value)def test_the_transformers_route_is_unchanged(self):from soup_cli.config.loader import load_config_from_stringdiff --git a/tests/test_mlx_simpo.py b/tests/test_mlx_simpo.pynew file mode 100644index 00000000..7f9fdd0f--- /dev/null+++ b/tests/test_mlx_simpo.py@@ -0,0 +1,784 @@+"""Native SimPO on MLX (#1999).++The objective is pinned to TRL 0.29.1's ``experimental/cpo/cpo_trainer.py`` with+``loss_type="simpo"``, which Soup's transformers backend runs for ``task: simpo``+(``trainer/simpo.py`` forwards ``simpo_gamma`` and ``cpo_alpha`` and no beta, so+``CPOConfig.beta``'s default applies). Layers:++- tokenization against TRL's own ``CPOTrainer.tokenize_row`` on a real+ ``PreTrainedTokenizerFast`` (needs ``trl`` / ``transformers``, no MLX);+- an independent float64 NumPy oracle, shown (without MLX) to be moved by every+ wrong objective the issue lists; the MLX loss, its terms and its gradients+ against that oracle and against TRL's own ``get_batch_loss_metrics`` on the+ same logits (MLX; TRL parts need torch);+- training, saving and a fresh-process reload on a tiny random Llama (needs+ MLX and mlx-lm). No downloads.+"""++from __future__ import annotations++import ast+import json+import math+import subprocess+import sys+import textwrap+from pathlib import Path++import numpy as np+import pytest++from tests.test_mlx_dpo import _strip_ansi+from tests.test_mlx_orpo import (+ CharTokenizer,+ MergingTokenizer,+ _decode,+ _Fixed,+ _flat,+ _rows,+ _tiny_llama_args,+)++BETA = 0.1 # CPOConfig.beta's default in TRL 0.29.1, pinned below.+++def _build(row, tokenizer=None, max_length=256, max_prompt_length=128):+ from soup_cli.trainer.mlx_orpo import build_orpo_example++ return build_orpo_example(+ row, tokenizer or CharTokenizer(), max_length=max_length,+ max_prompt_length=max_prompt_length,+ )+++# --------------------------------------------------------------------------+# TRL facts this objective rests on (no MLX)+# --------------------------------------------------------------------------+++class TestTrlFacts:+ def test_the_effective_beta_is_cpoconfig_default(self):+ pytest.importorskip("trl")+ from trl.experimental.cpo import CPOConfig++ from soup_cli.trainer.mlx_simpo import SIMPO_BETA++ fields = CPOConfig.__dataclass_fields__+ assert fields["beta"].default == SIMPO_BETA == BETA+ assert fields["label_smoothing"].default == 0.0+ assert fields["alpha"].default == 0.0 # AlphaPO's reward shaping is off++ def test_the_transformers_simpo_path_passes_no_beta(self):+ """Soup's transformers SimPO builds CPOConfig without beta or label+ smoothing, so the default above is what it trains with."""+ import soup_cli.trainer.simpo as simpo++ tree = ast.parse(Path(simpo.__file__).read_text(encoding="utf-8"))+ calls = [+ node for node in ast.walk(tree)+ if isinstance(node, ast.Call) and getattr(node.func, "id", "") == "cpo_config_cls"+ ]+ assert len(calls) == 1+ names = {kw.arg for kw in calls[0].keywords}+ assert {"loss_type", "simpo_gamma", "cpo_alpha"} <= names+ assert not names & {"beta", "label_smoothing", "alpha"}+++# --------------------------------------------------------------------------+# Tokenization: ORPO's, which is TRL's CPO tokenize_row too+# --------------------------------------------------------------------------+++class TestTokenization:+ def test_text_row_gets_bos_and_eos(self):+ ex = _build({"prompt": "Q:", "chosen": " yes", "rejected": " no"})+ assert _decode(ex.chosen_ids) == "^Q: yes$"+ assert ex.chosen_prompt_len == 3++ def test_a_merged_last_prompt_token_moves_the_answer_start(self):+ ex = _build({"prompt": "zx", "chosen": "y1", "rejected": "q2"}, MergingTokenizer())+ assert ex.chosen_ids[ex.chosen_prompt_len] == MergingTokenizer.MERGED++ def test_the_trainer_caps_the_prompt_keeping_its_end(self):+ from soup_cli.config.loader import load_config_from_string+ from soup_cli.trainer.mlx_simpo import MLXSimPOTrainerWrapper++ wrapper = MLXSimPOTrainerWrapper(load_config_from_string(_SIMPO_YAML))+ wrapper.config.data.max_length = 40+ wrapper.tokenizer = CharTokenizer()+ (example,), ties = wrapper._examples(+ [{"prompt": "S" * 60 + "TAIL", "chosen": "a", "rejected": "b"}]+ )+ assert ties == 0+ assert example.chosen_prompt_len == 20+ assert _decode(example.chosen_ids[: example.chosen_prompt_len]).endswith("TAIL")++ def test_no_room_for_an_answer_is_refused(self):+ from soup_cli.trainer.mlx_dpo import PreferenceRowError++ with pytest.raises(PreferenceRowError, match="no answer token fits"):+ _build({"prompt": "Q" * 30, "chosen": "a", "rejected": "b"},+ max_length=10, max_prompt_length=10)++ def test_a_tie_is_kept(self):+ """TRL trains a tie: the preference term is the constant softplus(gamma)."""+ from soup_cli.config.loader import load_config_from_string+ from soup_cli.trainer.mlx_simpo import MLXSimPOTrainerWrapper++ wrapper = MLXSimPOTrainerWrapper(load_config_from_string(_SIMPO_YAML))+ wrapper.tokenizer = CharTokenizer()+ examples, ties = wrapper._examples([{"prompt": "Q", "chosen": "s", "rejected": "s"}])+ assert (len(examples), ties) == (1, 0)+++class TestTokensMatchTrl:+ """The same rows through TRL 0.29.1's CPOTrainer.tokenize_row."""++ def _trl(self, tokenizer, row, max_length):+ import types++ from trl.data_utils import maybe_apply_chat_template+ from trl.experimental.cpo.cpo_trainer import CPOTrainer++ stub = types.SimpleNamespace(+ processing_class=tokenizer, is_encoder_decoder=False, max_length=max_length,+ )+ stub.build_tokenized_answer = types.MethodType(CPOTrainer.build_tokenized_answer, stub)+ return CPOTrainer.tokenize_row(stub, maybe_apply_chat_template(row, tokenizer))++ @pytest.mark.parametrize("row", [+ {"prompt": "Q: what is two plus two", "chosen": " four", "rejected": " five"},+ {"prompt": [{"role": "user", "content": "what is two plus two"}],+ "chosen": [{"role": "assistant", "content": "it is four"}],+ "rejected": [{"role": "assistant", "content": "five"}]},+ ])+ def test_ids_and_label_spans_match(self, tmp_path, row):+ from tests.test_mlx_orpo import TestTokensMatchTrl as Orpo++ tokenizer = Orpo()._tokenizer(tmp_path)+ trl = self._trl(tokenizer, row, max_length=64)+ ours = _build(row, tokenizer, max_length=64, max_prompt_length=32)+ for side in ("chosen", "rejected"):+ ids = getattr(ours, f"{side}_ids")+ span = getattr(ours, f"{side}_prompt_len")+ assert ids == list(trl[f"{side}_input_ids"])+ # CPO's labels mask the prompt: what is scored and what the NLL+ # covers is exactly the completion.+ assert span == sum(1 for x in trl[f"{side}_labels"] if x == -100)+ assert list(trl[f"{side}_labels"][span:]) == ids[span:]+++# --------------------------------------------------------------------------+# The objective against a float64 NumPy oracle+# --------------------------------------------------------------------------+++def _np_terms(logits, tokens, completion, sequence, *, mean=True, nll="token_mean",+ nll_mask="completion"):+ shifted = logits[:, :-1] - logits[:, :-1].max(-1, keepdims=True)+ logsm = shifted - np.log(np.exp(shifted).sum(-1, keepdims=True))+ lp = np.take_along_axis(logsm, tokens[:, 1:][..., None], -1)[..., 0]+ comp = completion[:, 1:].astype(np.float64)+ scores = (lp * comp).sum(-1) / (comp.sum(-1) if mean else 1.0)+ half = tokens.shape[0] // 2+ m = (completion if nll_mask == "completion" else sequence)[:half, 1:].astype(np.float64)+ if nll == "token_mean":+ value = -(lp[:half] * m).sum() / m.sum()+ else: # mean of per-row means+ value = (-(lp[:half] * m).sum(-1) / m.sum(-1)).mean()+ return scores[:half], scores[half:], value+++def _np_loss(a, b, nll, gamma, alpha, *, beta=BETA, margin="outside", use_nll=True):+ z = {+ "outside": beta * (a - b) - gamma,+ "inside": beta * ((a - b) - gamma),+ "none": beta * (a - b),+ }[margin]+ pref = np.logaddexp(0.0, -z) # -logsigmoid(z)+ return pref.mean() + (alpha * nll if use_nll else 0.0)+++def _fixture(seed=0, vocab=12):+ """Two pairs, unequal prompt and completion lengths, right padding."""+ from soup_cli.trainer.mlx_orpo import OrpoExample, collate_orpo++ examples = [+ OrpoExample([1, 5, 6, 7, 8, 9], 3, [1, 5, 6, 4, 2], 3),+ OrpoExample([1, 3, 8, 2], 2, [1, 3, 9, 10, 11, 2], 2),+ ]+ tokens, completion, sequence = (np.array(x) for x in collate_orpo(examples))+ rng = np.random.default_rng(seed)+ logits = rng.normal(0.0, 2.0, size=(tokens.shape[0], tokens.shape[1], vocab))+ return logits, tokens, completion, sequence+++def _oracle(logits, tokens, comp, seq, gamma, alpha):+ return _np_loss(*_np_terms(logits, tokens, comp, seq), gamma, alpha)+++#: (gamma, cpo_alpha), varied independently.+_SETTINGS = [(0.5, 1.0), (1.3, 1.0), (0.5, 0.25), (0.0, 2.0)]+++class TestTheOracleSeesEveryListedMutation:+ """No MLX: the fixture is sensitive to every wrong objective the issue names,+ so an implementation making any of them fails ``test_terms_and_total``."""++ @pytest.mark.parametrize("mutation", [+ "summed_scores", "reversed_pair", "no_margin", "margin_inside", "no_nll",+ "nll_row_means", "nll_over_sequence",+ ])+ def test_each_mutation_moves_the_loss(self, mutation):+ logits, tokens, comp, seq = _fixture()+ gamma, alpha = 0.5, 1.0+ right = _oracle(logits, tokens, comp, seq, gamma, alpha)+ a, b, nll = _np_terms(+ logits, tokens, comp, seq, mean=mutation != "summed_scores",+ nll="row_means" if mutation == "nll_row_means" else "token_mean",+ nll_mask="sequence" if mutation == "nll_over_sequence" else "completion",+ )+ if mutation == "reversed_pair":+ a, b = b, a+ margin = {"no_margin": "none", "margin_inside": "inside"}.get(mutation, "outside")+ wrong = _np_loss(a, b, nll, gamma, alpha, margin=margin, use_nll=mutation != "no_nll")+ assert abs(wrong - right) > 1e-3++ def test_gamma_and_alpha_move_the_loss_independently(self):+ logits, tokens, comp, seq = _fixture()+ values = {s: _oracle(logits, tokens, comp, seq, *s) for s in _SETTINGS}+ assert len({round(v, 6) for v in values.values()}) == len(_SETTINGS)+++def _mlx(logits, tokens, completion, gamma, alpha, **kw):+ mx = pytest.importorskip("mlx.core")+ from soup_cli.trainer.mlx_simpo import simpo_loss, simpo_terms++ model = _Fixed(logits)+ a, b, nll = simpo_terms(model, mx.array(tokens), mx.array(completion), **kw)+ loss, metrics = simpo_loss(a, b, nll, beta=BETA, gamma=gamma, cpo_alpha=alpha)+ return np.array(a), np.array(b), float(nll.item()), float(loss.item()), metrics+++class TestTheLossMatchesTheOracle:+ @pytest.mark.parametrize("gamma, alpha", _SETTINGS)+ def test_terms_and_total(self, gamma, alpha):+ logits, tokens, comp, seq = _fixture()+ a, b, nll, loss, metrics = _mlx(logits, tokens, comp, gamma, alpha)+ ra, rb, rnll = _np_terms(logits, tokens, comp, seq)+ np.testing.assert_allclose(a, ra, rtol=1e-5)+ np.testing.assert_allclose(b, rb, rtol=1e-5)+ assert nll == pytest.approx(rnll, rel=1e-5)+ assert loss == pytest.approx(_np_loss(ra, rb, rnll, gamma, alpha), rel=1e-5)+ assert metrics["nll_loss"].item() == pytest.approx(rnll, rel=1e-5)+ # The preference term, logged on its own: the total minus cpo_alpha * NLL.+ assert metrics["preference_loss"].item() == pytest.approx(+ _np_loss(ra, rb, rnll, gamma, alpha) - alpha * rnll, rel=1e-5+ )+ assert metrics["reward_chosen"].item() == pytest.approx(BETA * ra.mean(), rel=1e-5)+ assert metrics["reward_rejected"].item() == pytest.approx(BETA * rb.mean(), rel=1e-5)+ assert metrics["reward_margin"].item() == pytest.approx(+ BETA * (ra - rb).mean(), rel=1e-5+ )+ assert metrics["reward_accuracy"].item() == pytest.approx((ra > rb).mean())+ assert metrics["logps_chosen"].item() == pytest.approx(ra.mean(), rel=1e-5)+ assert metrics["logps_rejected"].item() == pytest.approx(rb.mean(), rel=1e-5)++ def test_the_chunked_head_gives_the_same_terms(self):+ mx = pytest.importorskip("mlx.core")+ from soup_cli.trainer.mlx_simpo import simpo_terms++ logits, tokens, comp, _ = _fixture()+ model = _Fixed(logits)+ split = (lambda x: model.logits[:, : x.shape[1]], lambda h: h)+ full = simpo_terms(model, mx.array(tokens), mx.array(comp))+ chunked = simpo_terms(model, mx.array(tokens), mx.array(comp), head_split=split)+ for x, y in zip(full, chunked):+ assert mx.allclose(x, y, atol=1e-6).item()++ def test_prompt_and_padding_logits_move_nothing(self):+ """CPO masks the prompt out of the NLL as well as the scores."""+ logits, tokens, comp, seq = _fixture()+ a0, b0, nll0, loss0, _ = _mlx(logits, tokens, comp, 0.5, 1.0)+ bumped = logits.copy()+ not_scored = comp[:, 1:] == 0 # prompt targets and padding+ bumped[:, :-1][not_scored] += np.random.default_rng(3).normal(+ 0.0, 3.0, size=(int(not_scored.sum()), logits.shape[-1])+ )+ a1, b1, nll1, loss1, _ = _mlx(bumped, tokens, comp, 0.5, 1.0)+ np.testing.assert_allclose(a1, a0, rtol=1e-6)+ np.testing.assert_allclose(b1, b0, rtol=1e-6)+ assert nll1 == pytest.approx(nll0, rel=1e-6)+ assert loss1 == pytest.approx(loss0, rel=1e-6)++ @pytest.mark.parametrize("gamma, alpha", [(0.5, 1.0), (1.3, 0.25)])+ def test_gradients_match_finite_differences(self, gamma, alpha):+ mx = pytest.importorskip("mlx.core")+ nn = pytest.importorskip("mlx.nn")+ from soup_cli.trainer.mlx_simpo import simpo_loss, simpo_terms++ logits, tokens, comp, seq = _fixture(seed=1)+ model = _Fixed(logits)+ t, c = mx.array(tokens), mx.array(comp)++ def f(m):+ return simpo_loss(*simpo_terms(m, t, c), beta=BETA, gamma=gamma,+ cpo_alpha=alpha)[0]++ _, grads = nn.value_and_grad(model, f)(model)+ g = np.array(grads["logits"])+ rng = np.random.default_rng(5)+ eps = 1e-4+ scored = np.argwhere(comp[:, 1:] == 1)+ for i in range(12):+ if i % 2: # half the probes on positions the loss reads+ r, p = scored[rng.integers(0, len(scored))]+ idx = (r, p, rng.integers(0, logits.shape[2]))+ else:+ idx = (rng.integers(0, logits.shape[0]), rng.integers(0, logits.shape[1] - 1),+ rng.integers(0, logits.shape[2]))+ hi, lo = logits.copy(), logits.copy()+ hi[idx] += eps+ lo[idx] -= eps+ fd = (_oracle(hi, tokens, comp, seq, gamma, alpha)+ - _oracle(lo, tokens, comp, seq, gamma, alpha)) / (2 * eps)+ assert g[idx] == pytest.approx(fd, abs=2e-4, rel=2e-2), idx+++class TestTheLossMatchesTrlItself:+ """TRL 0.29.1's own ``CPOTrainer.get_batch_loss_metrics`` on the same logits."""++ def _trl(self, logits, tokens, comp, seq, gamma, alpha):+ import types++ torch = pytest.importorskip("torch")+ pytest.importorskip("trl")+ from trl.experimental.cpo.cpo_trainer import CPOTrainer++ half = tokens.shape[0] // 2+ labels = np.where(comp == 1, tokens, -100)+ batch = {}+ for side, rows in (("chosen", slice(0, half)), ("rejected", slice(half, None))):+ batch[f"{side}_input_ids"] = torch.tensor(tokens[rows])+ batch[f"{side}_attention_mask"] = torch.tensor(seq[rows])+ batch[f"{side}_labels"] = torch.tensor(labels[rows])+ leaf = torch.tensor(logits, dtype=torch.float64, requires_grad=True)++ def model(input_ids, **_):+ return types.SimpleNamespace(logits=leaf[:, : input_ids.shape[1]])++ stub = types.SimpleNamespace(+ beta=BETA, label_smoothing=0.0, loss_type="simpo", simpo_gamma=gamma,+ cpo_alpha=alpha, alpha=0.0, is_encoder_decoder=False, pad_token_id=0,+ aux_loss_enabled=False,+ accelerator=types.SimpleNamespace(device="cpu", gather_for_metrics=lambda x: x),+ concatenated_inputs=CPOTrainer.concatenated_inputs,+ get_batch_logps=CPOTrainer.get_batch_logps,+ )+ for name in ("concatenated_forward", "cpo_loss", "get_batch_loss_metrics"):+ setattr(stub, name, types.MethodType(getattr(CPOTrainer, name), stub))+ loss, metrics = stub.get_batch_loss_metrics(model, batch)+ loss.backward()+ return loss.item(), metrics, leaf.grad.numpy()++ @pytest.mark.parametrize("gamma, alpha", _SETTINGS)+ def test_loss_metrics_and_gradient(self, gamma, alpha):+ mx = pytest.importorskip("mlx.core")+ nn = pytest.importorskip("mlx.nn")+ from soup_cli.trainer.mlx_simpo import simpo_loss, simpo_terms++ logits, tokens, comp, seq = _fixture(seed=2)+ trl_loss, trl_metrics, trl_grad = self._trl(logits, tokens, comp, seq, gamma, alpha)+ model = _Fixed(logits)+ t, c = mx.array(tokens), mx.array(comp)++ def f(m):+ return simpo_loss(*simpo_terms(m, t, c), beta=BETA, gamma=gamma, cpo_alpha=alpha)++ (loss, metrics), grads = nn.value_and_grad(model, f)(model)+ assert loss.item() == pytest.approx(trl_loss, rel=1e-5)+ for ours, theirs in (+ ("nll_loss", "nll_loss"), ("reward_chosen", "rewards/chosen"),+ ("reward_rejected", "rewards/rejected"), ("reward_accuracy", "rewards/accuracies"),+ ("reward_margin", "rewards/margins"), ("logps_chosen", "logps/chosen"),+ ("logps_rejected", "logps/rejected"),+ ):+ assert metrics[ours].item() == pytest.approx(trl_metrics[theirs], rel=1e-5,+ abs=1e-7), ours+ np.testing.assert_allclose(np.array(grads["logits"]), trl_grad, atol=2e-6)+++class TestTheOracleMatchesTrl:+ """No MLX: the NumPy oracle itself against TRL's ``get_batch_loss_metrics``,+ so a CI runner without MLX still checks what the MLX tests compare to."""++ @pytest.mark.parametrize("gamma, alpha", _SETTINGS)+ def test_loss_terms_and_gradient(self, gamma, alpha):+ logits, tokens, comp, seq = _fixture(seed=2)+ trl_loss, trl_metrics, trl_grad = TestTheLossMatchesTrlItself()._trl(+ logits, tokens, comp, seq, gamma, alpha+ )+ a, b, nll = _np_terms(logits, tokens, comp, seq)+ assert _np_loss(a, b, nll, gamma, alpha) == pytest.approx(trl_loss, rel=1e-9)+ assert nll == pytest.approx(trl_metrics["nll_loss"], rel=1e-9)+ assert a.mean() == pytest.approx(trl_metrics["logps/chosen"], rel=1e-9)+ assert b.mean() == pytest.approx(trl_metrics["logps/rejected"], rel=1e-9)+ assert BETA * (a - b).mean() == pytest.approx(trl_metrics["rewards/margins"], rel=1e-9)+ rng = np.random.default_rng(7)+ eps = 1e-5+ scored = np.argwhere(comp[:, 1:] == 1)+ assert np.abs(trl_grad).max() > 1e-3+ for _ in range(8): # positions the loss reads+ r, p = scored[rng.integers(0, len(scored))]+ idx = (r, p, rng.integers(0, logits.shape[2]))+ hi, lo = logits.copy(), logits.copy()+ hi[idx] += eps+ lo[idx] -= eps+ fd = (_oracle(hi, tokens, comp, seq, gamma, alpha)+ - _oracle(lo, tokens, comp, seq, gamma, alpha)) / (2 * eps)+ assert fd == pytest.approx(trl_grad[idx], abs=1e-7, rel=1e-5), idx+++class TestEdgeValues:+ def _loss(self, a, b, nll=0.5, gamma=0.5, alpha=1.0):+ mx = pytest.importorskip("mlx.core")+ from soup_cli.trainer.mlx_simpo import simpo_loss++ return simpo_loss(mx.array(a), mx.array(b), mx.array(nll), beta=BETA, gamma=gamma,+ cpo_alpha=alpha)++ def test_a_tie_is_softplus_gamma_plus_the_nll(self):+ loss, metrics = self._loss([-1.3, -0.2], [-1.3, -0.2], gamma=0.7, alpha=0.5)+ assert loss.item() == pytest.approx(math.log1p(math.exp(0.7)) + 0.25, rel=1e-6)+ assert metrics["reward_accuracy"].item() == 0.0 # strict, as TRL's `>`++ @pytest.mark.parametrize("a, b", [([-50.0], [-0.01]), ([-0.01], [-500.0])])+ def test_extreme_scores_are_finite(self, a, b):+ loss, _ = self._loss(a, b)+ expected = _np_loss(np.array(a), np.array(b), 0.5, 0.5, 1.0)+ assert loss.item() == pytest.approx(expected, rel=1e-5)++ def test_nan_input_is_not_a_quiet_zero(self):+ loss, _ = self._loss([float("nan")], [-1.0])+ assert math.isnan(loss.item())+++# --------------------------------------------------------------------------+# Training on a tiny random Llama+# --------------------------------------------------------------------------++_SIMPO_YAML = """+base: tiny-base+task: simpo+backend: mlx+data:+ train: ./x.jsonl+ format: dpo+ max_length: 128+training:+ epochs: 1+ lr: 5e-3+ simpo_gamma: 0.5+ quantization: none+ lora: {r: 4, alpha: 8, dropout: 0.0}+output: ./output+"""+++def _wrapper(tmp_path, monkeypatch, model_seed=0, **training):+ mx = pytest.importorskip("mlx.core")+ pytest.importorskip("mlx_lm")+ from mlx_lm.models import llama++ from soup_cli.config.loader import load_config_from_string+ from soup_cli.trainer.mlx_simpo import MLXSimPOTrainerWrapper++ tmp_path.mkdir(parents=True, exist_ok=True)+ monkeypatch.chdir(tmp_path)+ extra = "".join(f" {k}: {v}\n" for k, v in training.items())+ text = _SIMPO_YAML.replace(" simpo_gamma: 0.5\n", " simpo_gamma: 0.5\n" + extra)+ wrapper = MLXSimPOTrainerWrapper(load_config_from_string(text))+ mx.random.seed(model_seed)+ wrapper.model = llama.Model(_tiny_llama_args())+ wrapper.tokenizer = CharTokenizer()+ wrapper._dataset = {"train": _rows(), "val": _rows(2, prefix="v")}+ return wrapper+++class TestTraining:+ def test_lora_trains_and_the_base_stays_frozen(self, tmp_path, monkeypatch):+ wrapper = _wrapper(tmp_path, monkeypatch, batch_size=2)+ before = _flat(wrapper.model.parameters())+ result = wrapper.train()+ assert result["losses"] and all(math.isfinite(x) for x in result["losses"])+ after = _flat(wrapper.model.parameters())+ lora_b = [v for k, v in after.items() if k.endswith("lora_b")]+ assert lora_b and any(np.abs(v).sum() > 0 for v in lora_b)+ for name, value in before.items():+ moved = name.replace(".weight", ".linear.weight")+ key = name if name in after else moved+ np.testing.assert_array_equal(after[key], value, err_msg=name)++ def test_the_preference_is_learned_and_the_metadata_written(self, tmp_path, monkeypatch):+ wrapper = _wrapper(tmp_path, monkeypatch, epochs=6, batch_size=2, warmup_ratio=0.0,+ cpo_alpha=0.5)+ result = wrapper.train()+ assert all(math.isfinite(x) for x in result["losses"])+ assert result["val_reward_accuracy"] == pytest.approx(1.0)+ assert result["val_reward_margin"] > 0+ assert math.isfinite(result["val_nll_loss"])+ assert result["skipped_identical_pairs"] == 0+ meta = json.loads((tmp_path / "output" / "adapter_config.json").read_text())+ assert meta["task"] == "simpo" and meta["fine_tune_type"] == "lora"+ assert (meta["simpo_beta"], meta["simpo_gamma"], meta["cpo_alpha"]) == (0.1, 0.5, 0.5)+ assert not list((tmp_path / "output").glob("*.partial*"))++ def test_training_moves_the_held_out_margin_against_the_untrained_model(+ self, tmp_path, monkeypatch+ ):+ from soup_cli.trainer.mlx_simpo import simpo_loss, simpo_terms++ wrapper = _wrapper(tmp_path, monkeypatch, epochs=6, batch_size=2, warmup_ratio=0.0,+ gradient_accumulation_steps=1)+ examples, _ = wrapper._examples(_rows(2, prefix="v"), "val")++ def held_out():+ wrapper.model.eval()+ _, metrics = simpo_loss(*simpo_terms(wrapper.model, *wrapper._batch_arrays(examples)),+ beta=BETA, gamma=0.5, cpo_alpha=1.0)+ return metrics["reward_margin"].item()++ before = held_out()+ wrapper.train()+ assert held_out() > before + 0.1++ def test_two_micro_batches_equal_one_batch_after_an_sgd_update(self, tmp_path, monkeypatch):+ from soup_cli.trainer import mlx_dpo++ def run(path, batch_size, accum, plan):+ monkeypatch.setattr(mlx_dpo, "plan_batches", lambda *a, **k: [list(b) for b in plan])+ wrapper = _wrapper(path, monkeypatch, batch_size=batch_size,+ gradient_accumulation_steps=accum, optimizer="sgd",+ scheduler="constant", warmup_ratio=0.0, seed=4)+ result = wrapper.train()+ return result, _flat(wrapper.model.trainable_parameters())++ one, w1 = run(tmp_path / "a", 2, 1, [[0, 1]])+ two, w2 = run(tmp_path / "b", 1, 2, [[0], [1]])+ assert len(one["losses"]) == len(two["losses"]) == 1+ assert one["losses"][0] == pytest.approx(two["losses"][0], rel=1e-5)+ # Equal-length chosen completions, so the batch's token-mean NLL and+ # the mean of two single-pair NLLs coincide.+ for name in w1:+ np.testing.assert_allclose(w1[name], w2[name], atol=1e-6, err_msg=name)++ def test_a_trailing_partial_batch_is_dropped_and_val_chunks_weigh_by_rows(+ self, tmp_path, monkeypatch+ ):+ mx = pytest.importorskip("mlx.core")++ wrapper = _wrapper(tmp_path, monkeypatch, batch_size=2, gradient_accumulation_steps=1)+ wrapper._dataset["train"] = _rows(7)+ wrapper._dataset["val"] = _rows(2, prefix="v") + [+ {"prompt": "w:", "chosen": " bad", "rejected": " good"}+ ]+ result = wrapper.train()+ assert result["total_steps"] == 3 # 7 rows, batch 2: the 7th never trains+ examples, _ = wrapper._examples(wrapper._dataset["val"], "val")+ wrapper.model.eval()+ per_row = [+ wrapper._loss(wrapper.model, wrapper._batch_arrays([e]), "val", mx.arange(1))[0].item()+ for e in examples+ ]+ assert result["val_loss"] == pytest.approx(sum(per_row) / 3, rel=1e-4)++ def test_a_nonfinite_loss_stops_the_run(self, tmp_path, monkeypatch):+ mx = pytest.importorskip("mlx.core")+ from soup_cli.trainer import mlx_simpo++ real = mlx_simpo.simpo_terms++ def poisoned(*args, **kwargs):+ a, b, nll = real(*args, **kwargs)+ return a * mx.array(float("nan")), b, nll++ monkeypatch.setattr(mlx_simpo, "simpo_terms", poisoned)+ wrapper = _wrapper(tmp_path, monkeypatch, batch_size=2)+ with pytest.raises(FloatingPointError, match="MLX SimPO: nonfinite loss"):+ wrapper.train()+++class TestResume:+ def _first_run(self, tmp_path, monkeypatch):+ wrapper = _wrapper(tmp_path / "a", monkeypatch, batch_size=2, epochs=3, seed=1)+ fresh = wrapper.train()+ return tmp_path / "a" / "output" / "adapters.safetensors", fresh++ def test_resume_from_its_own_adapter(self, tmp_path, monkeypatch):+ checkpoint, fresh = self._first_run(tmp_path, monkeypatch)+ resumed = _wrapper(tmp_path / "b", monkeypatch, batch_size=2, seed=1)+ result = resumed.train(resume_from_checkpoint=str(checkpoint))+ assert result["losses"][0] < fresh["losses"][0]++ @pytest.mark.parametrize("change, match", [+ ({"task": "orpo"}, "MLX SimPO: .* task='orpo', not 'simpo'"),+ ({"task": "dpo"}, "task='dpo'"),+ ({"model": "other-base"}, "belongs to base"),+ (None, "MLX SimPO: no adapter_config.json"),+ ])+ def test_mismatched_or_missing_metadata_is_refused(self, tmp_path, monkeypatch, change, match):+ checkpoint, _ = self._first_run(tmp_path, monkeypatch)+ meta = checkpoint.parent / "adapter_config.json"+ if change is None:+ meta.unlink()+ else:+ meta.write_text(json.dumps({**json.loads(meta.read_text()), **change}))+ resumed = _wrapper(tmp_path / "b", monkeypatch, batch_size=2, seed=1)+ with pytest.raises(ValueError, match=match):+ resumed.train(resume_from_checkpoint=str(checkpoint))+++class TestFreshProcessReload:+ def test_logits_reproduce_and_the_adapter_has_an_effect(self, tmp_path, monkeypatch):+ mx = pytest.importorskip("mlx.core")++ wrapper = _wrapper(tmp_path, monkeypatch, batch_size=2, epochs=3)+ base_path = tmp_path / "base.safetensors"+ wrapper.model.save_weights(str(base_path))+ wrapper.train()+ examples, _ = wrapper._examples(_rows(2, prefix="v"), "val")+ tokens, _ = wrapper._batch_arrays(examples)+ wrapper.model.eval()+ logits = wrapper.model(tokens).astype(mx.float32)+ mx.eval(logits)+ np.save(tmp_path / "tokens.npy", np.array(tokens))++ script = textwrap.dedent(f"""+ import sys+ import numpy as np+ import mlx.core as mx+ from mlx_lm.models import llama+ from mlx_lm.tuner.utils import load_adapters+ args = llama.ModelArgs(model_type="llama", hidden_size=32, num_hidden_layers=2,+ intermediate_size=64, num_attention_heads=4, num_key_value_heads=2,+ rms_norm_eps=1e-5, vocab_size=200, tie_word_embeddings=True)+ model = llama.Model(args)+ model.load_weights({str(base_path)!r})+ if sys.argv[1] == "adapter":+ load_adapters(model, {str(tmp_path / "output")!r})+ model.eval()+ t = mx.array(np.load({str(tmp_path / "tokens.npy")!r}))+ np.save(sys.argv[2], np.array(model(t).astype(mx.float32)))+ """)++ def run(mode):+ out = tmp_path / f"{mode}.npy"+ subprocess.run([sys.executable, "-c", script, mode, str(out)], capture_output=True,+ text=True, check=True, timeout=300)+ return np.load(out)++ reloaded, base = run("adapter"), run("base")+ np.testing.assert_allclose(reloaded, np.array(logits), rtol=1e-4, atol=1e-5)+ assert np.abs(base - reloaded).max() > 1e-3+++# --------------------------------------------------------------------------+# Config and dispatch (no MLX)+# --------------------------------------------------------------------------+++class TestConfig:+ def _load(self, top="", training=""):+ """`training` lines replace the fixture's own line for the same key."""+ from soup_cli.config.loader import load_config_from_string++ text = _SIMPO_YAML.replace("backend: mlx\n", "backend: mlx\n" + top)+ for line in training.splitlines(keepends=True):+ key = line.split(":", 1)[0]+ own = [x for x in text.splitlines(keepends=True) if x.split(":", 1)[0] == key]+ text = text.replace(own[0], line) if own else text.replace(+ " simpo_gamma: 0.5\n", " simpo_gamma: 0.5\n" + line+ )+ return load_config_from_string(text)++ def test_the_configured_gamma_and_alpha_reach_the_loss(self, monkeypatch):+ """``simpo_gamma`` and ``cpo_alpha`` are the task's user-facing settings;+ beta is CPOConfig's default (review of #2097)."""+ from soup_cli.trainer import mlx_simpo++ cfg = self._load(training=" cpo_alpha: 0.3\n simpo_gamma: 1.3\n")+ wrapper = mlx_simpo.MLXSimPOTrainerWrapper(cfg)+ wrapper._head_split = None+ seen = {}+ monkeypatch.setattr(mlx_simpo, "simpo_terms", lambda *a, **k: (1.0, 2.0, 3.0))+ monkeypatch.setattr(mlx_simpo, "simpo_loss",+ lambda *a, **k: (seen.update(k), (0.0, {}))[1])+ wrapper._loss(object(), (None, None), "train", None)+ assert seen == {"beta": 0.1, "gamma": 1.3, "cpo_alpha": 0.3}++ def test_simpo_on_mlx_loads_and_is_an_mlx_task(self):+ from soup_cli.config.schema import MLX_TASKS++ cfg = self._load(training=" cpo_alpha: 0.3\n")+ assert (cfg.task, cfg.backend, cfg.training.simpo_gamma) == ("simpo", "mlx", 0.5)+ assert cfg.training.cpo_alpha == 0.3+ assert "simpo" in MLX_TASKS++ def test_simpo_dispatches_to_the_native_trainer(self):+ from soup_cli.trainer.mlx_routing import MLX_TRAINER_REGISTRY, get_mlx_trainer+ from soup_cli.trainer.mlx_simpo import MLXSimPOTrainerWrapper++ assert "simpo" in MLX_TRAINER_REGISTRY and "simpo" in MLX_TRAINER_REGISTRY.keys()+ assert get_mlx_trainer("simpo") is MLXSimPOTrainerWrapper++ @pytest.mark.parametrize("top, training, match", [+ ("", " quantization: 8bit\n", "quantization='8bit' is not supported by task='simpo'"),+ ("modality: vision\n", "", "modality='vision' is not supported by task='simpo'"),+ ("", " lora: {r: 0}\n", "lora.r must be at least 1 for task='simpo'"),+ ("", " loraplus_lr_ratio: 16\n",+ "training.loraplus_lr_ratio is not implemented on backend='mlx'"),+ ("", " cpo_alpha: 0\n", "cpo_alpha: Input should be greater than 0"),+ ])+ def test_unsupported_combinations_are_refused_at_load(self, top, training, match):+ with pytest.raises(Exception, match=match):+ self._load(top, training)++ def test_the_preference_alias_is_refused_naming_task_simpo(self):+ from soup_cli.config.loader import load_config_from_string++ alias = (_SIMPO_YAML.replace("task: simpo", "task: preference")+ .replace(" simpo_gamma: 0.5\n", " preference_loss: simpo\n"))+ with pytest.raises(Exception, match="task='preference' has no trainer") as info:+ load_config_from_string(alias)+ assert "Use task: simpo for the native MLX trainer." in str(info.value)++ def test_other_tasks_name_simpo_among_the_mlx_ones(self):+ from soup_cli.config.loader import load_config_from_string++ with pytest.raises(Exception, match="only ships SFT, DPO, ORPO and SimPO"):+ load_config_from_string(_SIMPO_YAML.replace("task: simpo", "task: ipo"))++ def test_the_transformers_route_is_unchanged(self):+ from soup_cli.config.loader import load_config_from_string++ cfg = load_config_from_string(_SIMPO_YAML.replace("backend: mlx", "backend: transformers"))+ assert cfg.backend == "transformers" and cfg.task == "simpo"++ def test_doctor_reports_ignored_settings(self):+ from soup_cli.config.backend_support import check_config++ cfg = self._load(training=" use_galore: true\n")+ assert {e.field for e in check_config(cfg)} == {"training.use_galore"}++ def test_the_ignored_settings_are_printed(self, monkeypatch):+ from io import StringIO++ from rich.console import Console++ from soup_cli.trainer import mlx_dpo+ from soup_cli.trainer.mlx_simpo import MLXSimPOTrainerWrapper++ buffer = StringIO()+ monkeypatch.setattr(mlx_dpo, "console", Console(file=buffer, width=400))+ MLXSimPOTrainerWrapper(self._load(training=" use_galore: true\n"))._check_unsupported()+ assert "MLX backend ignores: GaLore" in _strip_ansi(buffer.getvalue())