Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.
Python cliconsumer-gpudpofine-tuningggufhuggingfacellmllmopslocal-ailocal-llmloralow-vrammachine-learningollamapeftpythonpytorchqlorasfttransformers
feat(mlx): native KTO on Apple Silicon (#2000) (#2111)
dryn85-maxb815287f03b7
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex ac2efe11..3563c1f8 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -512,7 +512,7 @@ jobs:# loss, gradient and training-loop tests ran on no job at all.# tests/test_ci_mlx_smoke_files.py fails when a tests/test_mlx_*.py that# needs the real library is neither listed here nor exempted there.- run: pytest -o addopts= -q tests/test_rewind_mlx.py tests/test_mlx_dora_native.py tests/test_mlx_dpo.py tests/test_mlx_orpo.py tests/test_mlx_eval_steps.py tests/test_mlx_simpo.py tests/test_mlx_sft_seed.py+ run: pytest -o addopts= -q tests/test_rewind_mlx.py tests/test_mlx_dora_native.py tests/test_mlx_dpo.py tests/test_mlx_orpo.py tests/test_mlx_eval_steps.py tests/test_mlx_simpo.py tests/test_mlx_sft_seed.py tests/test_mlx_kto.pytorchao-contract:needs: plan# Not on a pull request, approval run included: it is not a required check.diff --git a/changelog.d/0.76.0/2111.added.md b/changelog.d/0.76.0/2111.added.mdnew file mode 100644index 00000000..614eb925--- /dev/null+++ b/changelog.d/0.76.0/2111.added.md@@ -0,0 +1,12 @@+- **KTO runs natively on Apple Silicon: `task: kto` with `backend: mlx` (#2000 by @dryn85-max in+ #2111).** It was refused at config load. Soup's own MLX loop now trains TRL 0.29.1's KTO+ objective with `training.kto_beta`: summed completion log-ratios against a reference+ precomputed once from the base model, and a detached KL estimate built from TRL's own+ mismatched rows. Batches run in dataset order, as in TRL. Rows that fit in `max_length` train+ on the same tokens on either backend. Labels are read by the loader's KTO converter, exactly+ as on the transformers backend. An empty completion, or a prompt that leaves no completion+ room, is refused before training. `batch_size` stays at least 2, and a run whose gradient+ accumulation cuts the last batches warns naming them. `--resume` refuses another task or+ base, and a nonfinite loss stops the run. `quantization: 8bit`, `lora.r: 0`, non-text+ modalities, DoRA and LoRA+ are refused on MLX. Try: `soup train --config soup.yaml` with+ `task: kto`, `backend: mlx` and `format: kto` data.diff --git a/docs/backends-and-ops.md b/docs/backends-and-ops.mdindex 675f823f..a9d4d541 100644--- a/docs/backends-and-ops.md+++ b/docs/backends-and-ops.md@@ -142,9 +142,9 @@ training:alpha: 32```-MLX backend supports SFT, DPO, ORPO and SimPO. Any other task with `backend: mlx` (`grpo`, `ipo`, `kto`, `preference`, ...) is refused when the config is loaded, with an error naming the task. Requires `mlx-lm >= 0.31.3`. Use `soup recipes search mlx` for ready-made Apple Silicon configs.+MLX backend supports SFT, DPO, ORPO, SimPO and KTO. Any other task with `backend: mlx` (`grpo`, `ipo`, `preference`, ...) is refused when the config is loaded, with an error naming the task. Requires `mlx-lm >= 0.31.3`. Use `soup recipes search mlx` for ready-made Apple Silicon configs.-`training.lora.rank_pattern` and `training.lora.alpha_pattern` are refused on `backend: mlx` for SFT, DPO, ORPO and SimPO ([#1996](https://github.com/MakazhanAlpamys/Soup/issues/1996)): the MLX trainer builds every adapter from the one scalar `lora.r` and `lora.alpha`, so per-module ranks and alphas would be dropped without a word. Remove them, or use `backend: transformers`, which applies them. An empty `{}` sets nothing and still loads.+`training.lora.rank_pattern` and `training.lora.alpha_pattern` are refused on `backend: mlx` for SFT, DPO, ORPO, SimPO and KTO ([#1996](https://github.com/MakazhanAlpamys/Soup/issues/1996)): the MLX trainer builds every adapter from the one scalar `lora.r` and `lora.alpha`, so per-module ranks and alphas would be dropped without a word. Remove them, or use `backend: transformers`, which applies them. An empty `{}` sets nothing and still loads.#### DPO on MLX@@ -286,11 +286,14 @@ That shows the objective moving, not alignment quality.- **Stops on a nonfinite loss** instead of training on NaN.- **LoRA dropout applies**, as in MLX DPO above; set `lora.dropout: 0` tomatch TRL's `disable_dropout`.-- **Reported:** loss, NLL, chosen and rejected rewards (`0.1 * logp`, as TRL- logs them), reward accuracy and margin, and the mean chosen and rejected- log-probabilities, per logged step and in the run result (`val_*` for the- validation split). TRL's mean-logit metrics are not computed: the vocabulary- head runs in chunks and the full logits are never held.+- **Reported:** the panel and `soup ui` show loss and validation loss. Without a+ live panel each logged step prints loss, NLL, reward accuracy and margin. The+ run result has, for the validation split, `val_preference_loss` (the SimPO+ term without the NLL, which is not on the per-step line), `val_nll_loss`, the+ chosen and rejected rewards (`0.1 * logp`, as TRL logs them), reward accuracy+ and margin, and the mean chosen and rejected log-probabilities. TRL's+ mean-logit metrics are not computed: the vocabulary head runs in chunks and+ the full logits are never held.- **Saved:** `adapters.safetensors` and `adapter_config.json` (with `task:simpo`, `simpo_beta`, `simpo_gamma`, `cpo_alpha` and the base model),loadable with `mlx_lm.load(base, adapter_path=output)`, written as MLX ORPO@@ -303,6 +306,112 @@ That shows the objective moving, not alignment quality.`backend: mlx` (use `task: simpo`). The other settings MLX DPO does not applyare not applied here either.+#### KTO on MLX++`task: kto` with `backend: mlx` runs the same loop and reference precompute as+MLX DPO with KTO's objective, matched to TRL 0.29.1's KTO trainer (the one the+transformers backend runs, with `training.kto_beta` as `beta` and every other+`KTOConfig` setting at its default):++- `r` is the policy minus the reference **summed** log-probability of the+ completion;+- `z = max(0, mean(policy_KL - reference_KL))` over the batch's KL rows,+ without gradient;+- a desirable row (`label: true`) loses `1 - sigmoid(kto_beta * (r - z))`, an+ undesirable one `1 - sigmoid(kto_beta * (z - r))`, each weighted 1.0+ (`desirable_weight` / `undesirable_weight` are not exposed), and the loss is+ the mean over every row of the batch. A batch with only one label trains on+ that label's rows alone.++Qualified on Qwen2.5-0.5B-Instruct in bf16 and 4-bit on an M4 Pro: one epoch on+1,200 unpaired UltraFeedback rows (`batch_size: 4`, accumulation 2) took about+9 minutes and 8.0-9.3 GB, and the KTO loss on 400 held-out rows fell from 0.500+to 0.448 (bf16) and 0.456 (4-bit). The desirable term fell while the undesirable+one grew, the usual one-epoch KTO shape; that shows the objective moving, not+alignment quality.++```yaml+base: mlx-community/Qwen2.5-0.5B-Instruct-4bit+task: kto+backend: mlx++data:+ train: ./kto.jsonl+ format: kto+ max_length: 1024++training:+ lr: 2e-5+ kto_beta: 0.1+ batch_size: 4+ gradient_accumulation_steps: 4+ gradient_checkpointing: true+ lora:+ r: 16+ alpha: 32+ dropout: 0+```++- **KL rows, as TRL builds them.** Each row's KL row is its own prompt with the+ completion of the previous row in its group of `batch_size` consecutive rows,+ the first row taking the last one's; a shorter last group rotates within+ itself, and a last group of one row is its own KL row. Batches run in+ dataset order, the last partial batch included, because TRL's KTO samples+ sequentially: each training batch is one KL group, so its `z` is estimated+ from the same completions it scores. With `gradient_accumulation_steps`+ above 1 the plan is cut to whole accumulation windows, as in MLX DPO and+ ORPO; in dataset order that always leaves the same final batches untrained+ (TRL instead updates on the partial window at each epoch's end), and the run+ warns before training, naming those batches and their rows. `training.seed` / `data_seed` do not+ reorder rows (on either backend); shuffle the file to mix labels.+- **Reference, precomputed.** The reference log-probs of every row and of its KL+ row come from the base model before the first update, as in MLX DPO+ (`--resume` included); the policy's KL log-probs are computed live, without+ gradient, at every step.+- **Tokens.** Rows are read by the `format: kto` converter and tokenized as+ TRL's KTO trainer does: rendered to text with the chat template (keeping the+ prefix the prompt and `prompt + completion` renderings share), the+ completion located even when the tokenizer merges the last prompt token+ into it, BOS prepended to the prompt and EOS appended to the completion when+ missing. A row longer than `data.max_length` loses the end of its completion;+ the prompt is never cut (TRL 0.29.1 has no prompt cap, and the transformers+ path adds none). A KL row is re-cut to its own prompt's room.+- **Labels** are read by the loader's `format: kto` converter, exactly as on the+ transformers backend (`true` / `false`, and the strings and numbers that+ converter accepts); MLX adds no reading of its own. Only a row handed to the+ trainer without the loader is checked for a real `true` / `false`.+- **Refused before any update:** a row with no `label`; an empty completion;+ and a row whose prompt leaves no room for one completion token in+ `data.max_length` (TRL keeps such a prompt whole and trains it past+ `max_length`). A refusal names the split and row index.+- **Batch size at least 2**, as on the transformers backend: `batch_size: 1` is+ refused at load and `batch_size: auto` runs at 2. A training split smaller+ than `batch_size` trains as one partial batch and one KL group, as in TRL;+ fewer than 2 rows is refused, because a row's KL completion is another+ row's.+- **Stops on a nonfinite loss** instead of training on it (TRL's `nanmean`+ drops a NaN row silently).+- **LoRA dropout applies**, as in MLX DPO above; set `lora.dropout: 0` to match+ TRL's `disable_dropout`. With dropout on, the KL forward draws its own mask.+- **Reported:** the panel and `soup ui` show loss and validation loss. Without a+ live panel each logged step also prints `kl`, `rewards/chosen` and+ `rewards/rejected` (`kto_beta * r` averaged per label, as TRL logs them;+ `n/a` for a label the step did not see). The run result has `val_kl`,+ `val_rewards_chosen`, `val_rewards_rejected`, `val_rewards_margins`,+ `val_logps_chosen` and `val_logps_rejected` for the validation split, which+ is scored in `batch_size` chunks in order. TRL's `logits/*` and `count/*`+ metrics are not reported.+- **Saved:** `adapters.safetensors` and `adapter_config.json` (with `task: kto`,+ `kto_beta`, the loss type, both weights, the KL group size and the base+ model), loadable with `mlx_lm.load(base, adapter_path=output)`, written as+ MLX ORPO writes them. `--resume` refuses a checkpoint without that metadata,+ from another task, or from another base, before any tensor loads.+- **Refused at load:** `quantization: 8bit` (use `none`, or a pre-quantized+ 4-bit base with `quantization: 4bit`), `training.lora.r: 0` (the reference is+ the untrained LoRA policy), non-text `modality`, DoRA, LoRA+ and+ `data.chat_template` (as for MLX DPO and ORPO). The other settings MLX DPO+ does not apply are not applied here either.+#### DoRA on MLXFor text SFT, set `training.lora.use_dora: true` to use MLX-LM's native@@ -364,8 +473,8 @@ what ran is recoverable from the output directory.### Transformers on MPS-The regular `backend: transformers` path can run more than MLX's SFT, DPO, ORPO and SimPO-surface. On a live MPS runtime that accepts bfloat16, Soup enables BF16 autocast+The regular `backend: transformers` path can run more than MLX's SFT, DPO, ORPO,+SimPO and KTO surface. On a live MPS runtime that accepts bfloat16, Soup enables BF16 autocastfor the hardware-validated text trainers: SFT, DPO, GRPO/RLVR, reward modelling,and PRM.The decision comes from a live MPS allocation probe rather than a macOS versiondiff --git a/docs/models.md b/docs/models.mdindex bc8956bb..0c480897 100644--- a/docs/models.md+++ b/docs/models.md@@ -175,7 +175,7 @@ spelling is unchanged. See [the README's install section](../README.md#1-install| `fast` | `pip install "soup-cli[fast]"` | Unsloth backend (2-5x faster, lower VRAM) || `vision` | `pip install "soup-cli[vision]"` | Vision / multimodal fine-tuning (Pillow) || `audio` | `pip install "soup-cli[audio]"` | Audio / speech fine-tuning (librosa, soundfile) |-| `mlx` | `pip install "soup-cli[mlx]"` | Standalone Apple Silicon SFT, DPO, ORPO and SimPO backend for local data; `[train]` is not required |+| `mlx` | `pip install "soup-cli[mlx]"` | Standalone Apple Silicon SFT, DPO, ORPO, SimPO and KTO backend for local data; `[train]` is not required || `qat` | `pip install "soup-cli[qat]"` | torchao: FP8 training (`quantization_aware: fp8`), FP8 attention, NVFP4, the `ao_adamw_*` optimizers and `soup export --format torchao` || `serve` | `pip install "soup-cli[serve]"` | Inference server (FastAPI + uvicorn) || `serve-fast` | `pip install "soup-cli[serve-fast]"` | vLLM inference backend (2-4x throughput) |diff --git a/docs/training.md b/docs/training.mdindex 887616a8..b9752740 100644--- a/docs/training.md+++ b/docs/training.md@@ -1878,6 +1878,9 @@ training:{"prompt": "What is 2+2?", "completion": "Fish", "label": false}```+On Apple Silicon, `backend: mlx` runs KTO natively with the same objective; see+[KTO on MLX](backends-and-ops.md#kto-on-mlx).+## ORPO Training (No Reference Model)diff --git a/src/soup_cli/config/backend_support.py b/src/soup_cli/config/backend_support.pyindex 00196638..e4d75134 100644--- a/src/soup_cli/config/backend_support.py+++ b/src/soup_cli/config/backend_support.py@@ -19,7 +19,7 @@ drift -- every entry must name a real field, and every dotted field name inthe truth; it can only stop it rotting, which is the failure that produced #749.Scope today: ``task=sft`` on ``backend=mlx`` and on ``backend=unsloth`` (#1124;-both apply to every modality), ``task=dpo``, ``task=orpo`` and ``task=simpo`` on+both apply to every modality), ``task=dpo``, ``task=orpo``, ``task=simpo`` and ``task=kto`` on``backend=mlx``, plus multimodal vision and audio on ``backend=transformers`` (#1156).Other combinations report nothing rather than guessing, which is the honest default fora table that has not been reviewed for them.@@ -212,12 +212,16 @@ _MLX_ORPO: tuple[SupportEntry, ...] = _MLX_DPO#: MLX SimPO is MLX ORPO's tokenization and MLX DPO's loop: the same fields._MLX_SIMPO: tuple[SupportEntry, ...] = _MLX_ORPO+#: MLX KTO runs MLX DPO's loop and reference precompute: the same fields.+_MLX_KTO: tuple[SupportEntry, ...] = _MLX_DPO+#: ``(task, backend, modality)`` -> the settings that combination does not honour.REGISTRY: dict[tuple[str, str, str], tuple[SupportEntry, ...]] = {("sft", "mlx", "text"): _MLX_SFT,("dpo", "mlx", "text"): _MLX_DPO,("orpo", "mlx", "text"): _MLX_ORPO,("simpo", "mlx", "text"): _MLX_SIMPO,+ ("kto", "mlx", "text"): _MLX_KTO,("sft", "unsloth", "text"): _UNSLOTH_SFT,("sft", "transformers", "vision"): _TRANSFORMERS_VISION_SFT,("sft", "transformers", "audio"): _TRANSFORMERS_AUDIO_SFT,@@ -271,6 +275,15 @@ TRAINER_MODULES: dict[tuple[str, str, str], tuple[str, ...]] = {"soup_cli/trainer/_trl_compat.py","soup_cli/trainer/loss_summary.py",),+ ("kto", "mlx", "text"): (+ "soup_cli/trainer/mlx_kto.py",+ "soup_cli/trainer/mlx_orpo.py",+ "soup_cli/trainer/mlx_dpo.py",+ "soup_cli/trainer/mlx_sft.py",+ "soup_cli/trainer/mlx_optim.py",+ "soup_cli/trainer/_trl_compat.py",+ "soup_cli/trainer/loss_summary.py",+ ),# utils/unsloth.py handles the loader path. trainer/sft.py::_setup_unsloth is not listed# here because listing sft.py would cause the drift guard to inspect transformers-only# precision reads; an AST test in test_issue1124 guards _setup_unsloth instead.diff --git a/src/soup_cli/config/schema.py b/src/soup_cli/config/schema.pyindex 31a30044..f25846ae 100644--- a/src/soup_cli/config/schema.py+++ b/src/soup_cli/config/schema.py@@ -5109,9 +5109,10 @@ UNSLOTH_SETUP_TASKS: frozenset[str] = frozenset({})#: The tasks with a trainer on ``backend: mlx``: SFT through mlx-lm's tuner and-#: DPO, ORPO and SimPO through Soup's own loop (``trainer/mlx_dpo.py``). Validators that defer-#: to :meth:`SoupConfig._validate_mlx_task_support` read this, not ``"sft"``.-MLX_TASKS: frozenset[str] = frozenset({"sft", "dpo", "orpo", "simpo"})+#: DPO, ORPO, SimPO and KTO through Soup's own loop (``trainer/mlx_dpo.py``).+#: Validators that defer to :meth:`SoupConfig._validate_mlx_task_support` read+#: this, not ``"sft"``.+MLX_TASKS: frozenset[str] = frozenset({"sft", "dpo", "orpo", "simpo", "kto"})#: #1497 — the tasks whose trainer reads ``training.freeze_layers`` /#: ``training.freeze_ratio``: ``SFTTrainerWrapper._setup_transformers``, which@@ -7036,8 +7037,9 @@ class SoupConfig(BaseModel):I/O on a real base. The same refusal now applies to every``task='kto'`` config, resident or streamed, before anything loads."""- if self.backend == "mlx":+ if self.backend == "mlx" and self.task not in MLX_TASKS:return self # _validate_mlx_task_support gives the more basic answer+ # MLX KTO (#2000) builds TRL's KL rows and keeps the same floor.tcfg = self.trainingif (self.task == "kto"@@ -8581,23 +8583,23 @@ class SoupConfig(BaseModel):@model_validator(mode="after")def _validate_mlx_task_support(self) -> "SoupConfig":- """MLX backend supports sft, dpo, orpo and simpo.+ """MLX backend supports sft, dpo, orpo, simpo and kto.- SFT runs through mlx-lm's tuner; DPO, ORPO and SimPO through Soup's own- loop in ``trainer/mlx_dpo.py`` / ``trainer/mlx_orpo.py`` /- ``trainer/mlx_simpo.py``. Every other task -- GRPO included, whose MLX- wrapper is still scaffolding -- is refused here rather than at- ``train()`` time.+ SFT runs through mlx-lm's tuner; DPO, ORPO, SimPO and KTO through Soup's+ own loop in ``trainer/mlx_dpo.py`` / ``trainer/mlx_orpo.py`` /+ ``trainer/mlx_simpo.py`` / ``trainer/mlx_kto.py``. Every other task --+ GRPO included, whose MLX wrapper is still scaffolding -- is refused here+ rather than at ``train()`` time."""if self.backend != "mlx":return self- if self.task in ("dpo", "orpo", "simpo"):+ if self.task in ("dpo", "orpo", "simpo", "kto"):# The native preference loops tokenize text and chat rows only and- # train a LoRA adapter. DPO's reference log-probs come from the- # policy before training, which is the base model only because a- # LoRA adapter's B starts at zero; ORPO and SimPO train LoRA only- # (no full fine-tune on MLX yet).- name = {"dpo": "DPO", "orpo": "ORPO", "simpo": "SimPO"}[self.task]+ # train a LoRA adapter. DPO's and KTO's reference log-probs come+ # from the policy before training, which is the base model only+ # because a LoRA adapter's B starts at zero; ORPO and SimPO train+ # LoRA only (no full fine-tune on MLX yet).+ name = {"dpo": "DPO", "orpo": "ORPO", "simpo": "SimPO", "kto": "KTO"}[self.task]if self.modality != "text":raise ValueError(f"modality='{self.modality}' is not supported by task='{self.task}' "@@ -8608,7 +8610,7 @@ class SoupConfig(BaseModel):why = ("the reference model is the untrained LoRA policy, which needs ""a LoRA adapter to exist"- if self.task == "dpo"+ if self.task in ("dpo", "kto")else f"MLX {name} trains a LoRA adapter; full fine-tuning is not ""implemented on backend='mlx'")@@ -8616,7 +8618,7 @@ class SoupConfig(BaseModel):f"training.lora.r must be at least 1 for task='{self.task}' on "f"backend='mlx': {why}.")- if self.task in ("orpo", "simpo") and self.training.quantization == "8bit":+ if self.task in ("orpo", "simpo", "kto") and self.training.quantization == "8bit":raise ValueError(f"training.quantization='8bit' is not supported by task='{self.task}' ""on backend='mlx': use quantization: none, or a pre-quantized "@@ -8630,8 +8632,8 @@ class SoupConfig(BaseModel):if self.task == "preference" and self.training.preference_loss in native:hint = f" Use task: {self.training.preference_loss} for the native MLX trainer."raise ValueError(- f"MLX backend only ships SFT, DPO, ORPO and SimPO; task='{self.task}' has no "- "trainer on backend='mlx'."+ f"MLX backend only ships SFT, DPO, ORPO, SimPO and KTO; task='{self.task}' has "+ "no trainer on backend='mlx'."+ (hint or f" Use backend=transformers for task={self.task}."))@@ -8667,9 +8669,10 @@ class SoupConfig(BaseModel):``backend: mlx``. Stop-gap: delete this validator (and``tests/test_mlx_lora_pattern_refusal.py``) when the MLX trainer honours them.- SFT, DPO, ORPO and SimPO all build the adapter in ``MLXSFTTrainerWrapper._apply_lora``- from one scalar ``lora.r`` and ``lora.alpha``, so a config that set a pattern- trained a uniform-rank adapter without a word. Not part of+ SFT, DPO, ORPO, SimPO and KTO all build the adapter in+ ``MLXSFTTrainerWrapper._apply_lora`` from one scalar ``lora.r`` and+ ``lora.alpha``, so a config that set a pattern trained a uniform-rank adapter+ without a word. Not part of:meth:`_validate_peft_variant_backend_and_quantization`, which returns earlyfor plain LoRA; defined after :meth:`_validate_mlx_task_support` (and the``lora.r=0`` check) on purpose, so a task MLX cannot run, or a config with nodiff --git a/src/soup_cli/trainer/mlx_dpo.py b/src/soup_cli/trainer/mlx_dpo.pyindex 60a811af..e3ca1ee1 100644--- a/src/soup_cli/trainer/mlx_dpo.py+++ b/src/soup_cli/trainer/mlx_dpo.py@@ -427,7 +427,10 @@ class MLXDPOTrainerWrapper(MLXSFTTrainerWrapper):self.model.eval()chosen, rejected = [], []total = len(examples)- console.print(f"[dim]MLX DPO: computing reference log-probs for {total} pair(s)[/]")+ console.print(+ f"[dim]MLX {self.objective}: computing reference log-probs for {total} "+ f"{self.reference_unit}[/]"+ )every = max(batch_size, (total // 10) // batch_size * batch_size)for start in range(0, total, batch_size):if start and start % every == 0:@@ -510,6 +513,33 @@ class MLXDPOTrainerWrapper(MLXSFTTrainerWrapper):"""``(tokens, completion mask)`` for the head-split check."""return self._to_arrays(examples)+ #: What one precomputed reference entry scores, for the progress message.+ reference_unit = "pair(s)"++ def _resolve_batch_size(self) -> int:+ """``training.batch_size`` as this loop runs it (``auto`` is 1 here)."""+ tcfg = self.config.training+ return int(tcfg.batch_size) if isinstance(tcfg.batch_size, int) else 1++ def _plan_batches(self, n_rows: int, batch_size: int, epochs: int, seed: int) -> list:+ """Every epoch's batches of example indices, in training order."""+ return plan_batches(n_rows, batch_size, epochs, seed)++ def _check_train_rows(self, n_rows: int, batch_size: int, ties: int) -> None:+ """Refuse a training split this objective cannot batch."""+ if n_rows < batch_size:+ raise ValueError(+ f"MLX {self.objective}: {n_rows} training pair(s) remain, fewer than "+ f"training.batch_size ({batch_size})"+ + (f", after skipping {ties} tie(s)" if ties else "")+ + f". Lower training.batch_size to at most {max(1, n_rows)} "+ "or add rows."+ )++ def _on_plan_cut(self, dropped: list, kept: int, accum: int) -> None:+ """Called with the batches an accumulation cut leaves untrained; the+ shuffled plans of DPO and ORPO drop different rows each run, silently."""+def _check_finite(self, loss: Any, split: str) -> None:import math as _math@@ -576,7 +606,7 @@ class MLXDPOTrainerWrapper(MLXSFTTrainerWrapper):output_dir = Path(cfg.output)output_dir.mkdir(parents=True, exist_ok=True)- batch_size = int(tcfg.batch_size) if isinstance(tcfg.batch_size, int) else 1+ batch_size = self._resolve_batch_size()accum = max(1, int(tcfg.gradient_accumulation_steps))train_examples, train_ties = self._examples(list(self._dataset.get("train", [])))@@ -587,22 +617,19 @@ class MLXDPOTrainerWrapper(MLXSFTTrainerWrapper):f"[yellow]MLX {name}: skipped {skipped} {name.lower()} row(s) with ""identical chosen and rejected (a tie carries no preference signal)[/]")- if len(train_examples) < batch_size:- raise ValueError(- f"MLX {name}: {len(train_examples)} training pair(s) remain, fewer than "- f"training.batch_size ({batch_size})"- + (f", after skipping {train_ties} tie(s)" if train_ties else "")- + f". Lower training.batch_size to at most {max(1, len(train_examples))} "- "or add rows."- )- plan = plan_batches(len(train_examples), batch_size, int(tcfg.epochs), data_seed)+ self._check_train_rows(len(train_examples), batch_size, train_ties)+ epoch_count = max(1, int(tcfg.epochs))+ plan = self._plan_batches(len(train_examples), batch_size, epoch_count, data_seed)+ batches_per_epoch = max(1, len(plan) // epoch_count)iters = len(plan)if accum > 1:# Like MLX SFT: no partial accumulation group is ever flushed.- iters = max(accum, iters - iters % accum)+ cut = max(accum, iters - iters % accum)+ if cut < iters:+ self._on_plan_cut(plan[cut:], cut, accum)+ iters = cutplan = (plan * math.ceil(iters / len(plan)))[:iters]total_updates = max(1, iters // accum)- batches_per_epoch = max(1, len(train_examples) // batch_size)self._apply_lora(self.model)if tcfg.gradient_checkpointing:diff --git a/src/soup_cli/trainer/mlx_kto.py b/src/soup_cli/trainer/mlx_kto.pynew file mode 100644index 00000000..c4372ab1--- /dev/null+++ b/src/soup_cli/trainer/mlx_kto.py@@ -0,0 +1,479 @@+"""MLX KTO trainer — unpaired preference alignment on Apple Silicon.++The objective is TRL 0.29.1's ``experimental/kto/kto_trainer.py``, the trainer+Soup's transformers backend runs for ``task: kto`` (``trainer/kto.py`` forwards+``kto_beta`` and ``max_length``; everything else stays at ``KTOConfig``'s+defaults: ``loss_type="kto"``, both class weights 1.0):++- ``r`` = policy minus reference **summed** completion log-probability+ (``get_batch_logps(..., average_log_prob=False)``);+- ``z`` = ``mean(policy_KL - reference_KL)`` over the batch's KL rows,+ detached, then clamped at 0 (``kto_loss``);+- desirable rows lose ``w_d * (1 - sigmoid(beta * (r - z)))``, undesirable rows+ ``w_u * (1 - sigmoid(beta * (z - r)))``; the loss is the mean over every row+ of the batch (TRL concatenates both classes, then takes ``nanmean``), so a+ class missing from a batch simply contributes nothing.++The KL rows are TRL's: each example's prompt with the completion of the+previous example in its group, the first taking the last (``_get_kl_dataset``,+mapped with ``batch_size=per_device_train_batch_size``; a last, shorter group+rotates within itself, and a group of one is its own KL row). TRL's KTO samples+batches **sequentially** (``_get_train_sampler`` returns a ``SequentialSampler``)+and keeps the last partial batch, so a training batch is exactly one KL group;+this trainer plans its batches the same way.++The loop, LoRA attach, optimizer plan, clipping, seeding, chunked head, adapter+writer and reference precompute are MLX DPO's+(:class:`~soup_cli.trainer.mlx_dpo.MLXDPOTrainerWrapper`): the reference+log-probs of every matched and KL row are computed once from the base model+before the first update (LoRA ``B`` starts at zero), and the policy's KL+log-probs are computed live, without gradient.++Tokenization is TRL's KTO ``_tokenize`` / ``_process_tokens``: rows are rendered+to text first (``apply_chat_template`` with TRL's common-prefix rule), the+completion is located with TRL's one-token merge rule, BOS is prepended to the+prompt and EOS appended to the completion when missing, and an over-long row+loses the end of its completion (the prompt is never cut; there is no prompt+cap in 0.29.1). A row whose completion keeps no token is refused, where TRL+would train the prompt plus a bare EOS (or slice past the prompt).+"""++from __future__ import annotations++from dataclasses import dataclass+from typing import Any++from rich.console import Console++from soup_cli.trainer.mlx_dpo import (+ MLXDPOTrainerWrapper,+ PreferenceRowError,+ sequence_logps,+)+from soup_cli.trainer.mlx_orpo import _common_prefix, _tokenized_answer++console = Console()++#: ``KTOConfig``'s defaults in TRL 0.29.1, which Soup's transformers path keeps.+KTO_LOSS_TYPE = "kto"+DESIRABLE_WEIGHT = 1.0+UNDESIRABLE_WEIGHT = 1.0++#: TRL refuses a per-device batch of 1: the KL row would be the matched row.+KTO_MIN_BATCH = 2+++@dataclass(frozen=True)+class KtoExample:+ """One tokenized row: its prompt, its completion and its KL completion.++ The KL row shares the prompt; its completion is another row's, re-cut to+ this prompt's room, as TRL's second ``_process_tokens`` pass does.+ """++ prompt_ids: list+ completion_ids: list+ kl_completion_ids: list+ label: bool+++# --------------------------------------------------------------------------+# Tokenization (TRL 0.29.1 KTO _tokenize / _process_tokens / _get_kl_dataset)+# --------------------------------------------------------------------------+++def _normalise_row(row: dict) -> dict:+ """The row in one layout via the loader's converter.++ Labels are read by the loader's KTO converter (``_convert_kto``) when+ ``load_dataset`` loads the file, exactly as on the transformers backend, so+ every row that reaches the trainer that way already has a bool label. The+ bool check below only guards rows handed to ``setup()`` without the loader:+ those are not given a second, different reading.+ """+ from soup_cli.data.formats import _convert_kto++ if "label" not in row:+ raise PreferenceRowError("the row has no 'label' (true for desirable, false for not)")+ label = row["label"]+ if not isinstance(label, bool):+ raise PreferenceRowError(+ f"label must be true or false, got {label!r} ({type(label).__name__}); "+ "rows that bypass the loader are not converted"+ )+ try:+ return _convert_kto(row)+ except (KeyError, TypeError, ValueError) as exc:+ raise PreferenceRowError(str(exc)) from exc+++def render_kto_text(row: dict, tokenizer: Any) -> tuple[str, str]:+ """``(prompt, completion)`` text, as ``trl.data_utils.apply_chat_template``.++ Text rows pass through. Conversational rows render the prompt with a+ generation prompt (or ``continue_final_message`` after an assistant turn),+ then ``prompt + completion``; the prompt is narrowed to its common prefix+ with that rendering and the completion is what follows it.+ """+ prompt, completion = row["prompt"], row["completion"]+ if not isinstance(prompt, list):+ return prompt, completion+ last_role = prompt[-1]["role"]+ if last_role in ("user", "tool"):+ kwargs = {"add_generation_prompt": True, "continue_final_message": False}+ elif last_role == "assistant":+ kwargs = {"add_generation_prompt": False, "continue_final_message": True}+ else:+ raise PreferenceRowError(f"the prompt ends with an unsupported role {last_role!r}")+ prompt_text = tokenizer.apply_chat_template(prompt, tokenize=False, **kwargs)+ full = tokenizer.apply_chat_template(prompt + completion, tokenize=False)+ prompt_text = _common_prefix(prompt_text, full)+ return prompt_text, full[len(prompt_text):]+++def process_kto_tokens(prompt_ids: list, answer_ids: list, tokenizer: Any,+ max_length: int) -> tuple[list, list, int]:+ """``(prompt_ids, completion_ids, answer tokens kept)`` after TRL's ``_process_tokens``.++ The room is ``max_length`` less one for a BOS and one for an EOS that are+ still to be added; a row past it loses the end of its answer. BOS is then+ prepended to the prompt and EOS appended to the (cut) answer when missing.+ A prompt longer than the room keeps no answer token here, where TRL's+ negative slice would keep some and train past ``max_length``;+ :func:`build_kto_examples` refuses such a matched row.+ """+ bos = getattr(tokenizer, "bos_token_id", None)+ eos = getattr(tokenizer, "eos_token_id", None)+ if eos is None:+ raise PreferenceRowError("the tokenizer has no eos_token_id to end the completion")+ prompt_ids, answer_ids = list(prompt_ids), list(answer_ids)+ limit = int(max_length)+ if prompt_ids and bos != prompt_ids[0]:+ limit -= 1+ if answer_ids and eos != answer_ids[-1]:+ limit -= 1+ if len(prompt_ids) + len(answer_ids) > limit:+ answer_ids = answer_ids[: max(0, limit - len(prompt_ids))]+ kept = len(answer_ids)+ if bos is not None and (not prompt_ids or bos != prompt_ids[0]):+ prompt_ids = [bos] + prompt_ids+ if not answer_ids or eos != answer_ids[-1]:+ answer_ids = answer_ids + [eos]+ return prompt_ids, answer_ids, kept+++def kl_partners(n_rows: int, group_size: int) -> list:+ """Which row's completion each row's KL row borrows.++ TRL's ``_get_kl_dataset`` rotates the completions of each mapped batch by+ one (``[last] + all[:-1]``) and ``Dataset.map(batched=True, batch_size=n)``+ hands it consecutive groups of ``n`` rows, the last one shorter when ``n``+ does not divide the dataset; a last group of one keeps its own completion.+ """+ size = max(1, int(group_size))+ partners = []+ for start in range(0, n_rows, size):+ stop = min(n_rows, start + size)+ partners.extend([stop - 1] + list(range(start, stop - 1)))+ return partners+++def build_kto_examples(rows: list, tokenizer: Any, *, max_length: int,+ group_size: int) -> list:+ """Tokenize ``{prompt, completion, label}`` rows and their KL rows like TRL 0.29.1.++ Raises :class:`PreferenceRowError` (prefixed ``row <index>:``) for a row+ with no usable label or no completion token, so a run never starts on+ shifted KL groups.+ """+ raw = []+ for index, row in enumerate(rows):+ try:+ norm = _normalise_row(row)+ prompt, completion = render_kto_text(norm, tokenizer)+ prompt_ids, answer_ids = _tokenized_answer(tokenizer, prompt, completion)+ if not answer_ids:+ raise PreferenceRowError(+ "the completion is empty: there is no completion token to score"+ )+ full_prompt, full_answer, kept = process_kto_tokens(+ prompt_ids, answer_ids, tokenizer, max_length+ )+ if not kept:+ raise PreferenceRowError(+ f"no completion token fits in data.max_length ({max_length}) after the "+ f"{len(full_prompt)}-token prompt"+ )+ except PreferenceRowError as exc:+ raise PreferenceRowError(f"row {index}: {exc}") from exc+ raw.append((full_prompt, full_answer, list(answer_ids), norm["label"]))++ examples = []+ for (prompt_ids, completion_ids, _, label), j in zip(raw, kl_partners(len(raw), group_size)):+ # TRL's KL pass re-processes this (BOS-prefixed) prompt with the+ # partner's uncut answer.+ _, kl_ids, _ = process_kto_tokens(prompt_ids, raw[j][2], tokenizer, max_length)+ examples.append(KtoExample(prompt_ids, completion_ids, kl_ids, label))+ return examples+++def collate_kto(examples: list) -> tuple[list, list]:+ """``(tokens, completion mask)``: the matched rows, then the KL rows,+ right-padded with 0. The mask marks the completion tokens whose+ log-probs are summed; padding is never read."""+ seqs, masks = [], []+ for side in ("completion_ids", "kl_completion_ids"):+ for ex in examples:+ completion = getattr(ex, side)+ seqs.append(list(ex.prompt_ids) + list(completion))+ masks.append([0] * len(ex.prompt_ids) + [1] * len(completion))+ width = max(len(s) for s in seqs)+ return ([s + [0] * (width - len(s)) for s in seqs],+ [m + [0] * (width - len(m)) for m in masks])+++def sequential_batches(n_rows: int, batch_size: int, epochs: int) -> list:+ """TRL's KTO batches: dataset order every epoch, the last partial batch kept+ (``SequentialSampler``, ``DataLoader(drop_last=False)``)."""+ size = max(1, int(batch_size))+ one = [list(range(i, min(n_rows, i + size))) for i in range(0, n_rows, size)]+ return [list(b) for _ in range(max(1, int(epochs))) for b in one]+++# --------------------------------------------------------------------------+# Scoring and loss+# --------------------------------------------------------------------------+++def kto_logps(model: Any, tokens: Any, mask: Any, n: int, *,+ head_split: Any = None) -> tuple[Any, Any]:+ """``(policy logps, policy KL logps)`` for a collated batch of ``n`` examples.++ Two forwards, as TRL runs them: the KL rows first and without gradient+ (``_compute_kl_logps`` under ``torch.no_grad``), then the matched rows.+ """+ import mlx.core as mx++ kl = mx.stop_gradient(+ sequence_logps(model, tokens[n:], mask[n:], head_split=head_split)+ )+ return sequence_logps(model, tokens[:n], mask[:n], head_split=head_split), kl+++def kto_loss(+ policy_logps: Any,+ policy_kl_logps: Any,+ reference_logps: Any,+ reference_kl_logps: Any,+ labels: Any,+ beta: float,+ *,+ desirable_weight: float = DESIRABLE_WEIGHT,+ undesirable_weight: float = UNDESIRABLE_WEIGHT,+) -> tuple[Any, dict]:+ """TRL 0.29.1's ``kto_loss`` with ``loss_type="kto"``, averaged over the batch.++ ``1 - sigmoid(x)`` is computed as ``sigmoid(-x)``. Metrics are sums over+ the batch divided by its row count (``chosen_share`` is the desirable+ fraction), so they stay finite for a single-label batch and average+ correctly over accumulation and validation chunks; a class mean is the+ ratio of its ``*_per_row`` metric to its share.+ """+ import mlx.core as mx++ labels = labels.astype(mx.bool_)+ kl = (policy_kl_logps - reference_kl_logps).astype(mx.float32).mean()+ z = mx.maximum(mx.stop_gradient(kl), 0.0)+ r = (policy_logps - reference_logps).astype(mx.float32)+ chosen = desirable_weight * mx.sigmoid(-beta * (r - z))+ rejected = undesirable_weight * mx.sigmoid(-beta * (z - r))+ loss = mx.where(labels, chosen, rejected).mean()+ rows = labels.size+ is_chosen = labels.astype(mx.float32)+ rewards = beta * mx.stop_gradient(r)+ logps = mx.stop_gradient(policy_logps).astype(mx.float32)+ metrics = {+ "kl": z,+ "chosen_share": is_chosen.mean(),+ "chosen_rewards_per_row": (rewards * is_chosen).sum() / rows,+ "rejected_rewards_per_row": (rewards * (1 - is_chosen)).sum() / rows,+ "chosen_logps_per_row": (logps * is_chosen).sum() / rows,+ "rejected_logps_per_row": (logps * (1 - is_chosen)).sum() / rows,+ }+ return loss, metrics+++def class_means(metrics: dict, prefix: str = "") -> dict:+ """TRL's logged ``rewards/chosen`` etc. from the per-row metrics: each class+ mean is its per-row sum over its share; a class with no rows is left out."""+ share = metrics[f"{prefix}chosen_share"]+ out = {}+ for side, part in (("chosen", share), ("rejected", 1.0 - share)):+ if part > 1e-12:+ for what in ("rewards", "logps"):+ out[f"{prefix}{what}_{side}"] = metrics[f"{prefix}{side}_{what}_per_row"] / part+ if f"{prefix}rewards_chosen" in out and f"{prefix}rewards_rejected" in out:+ out[f"{prefix}rewards_margins"] = (+ out[f"{prefix}rewards_chosen"] - out[f"{prefix}rewards_rejected"]+ )+ return out+++# --------------------------------------------------------------------------+# Trainer+# --------------------------------------------------------------------------+++class MLXKTOTrainerWrapper(MLXDPOTrainerWrapper):+ """KTO on MLX: MLX DPO's loop and reference precompute with KTO's rows and loss."""++ objective = "KTO"+ reference_unit = "row(s) and their KL rows"+ metric_names = (+ "kl", "chosen_share", "chosen_rewards_per_row", "rejected_rewards_per_row",+ "chosen_logps_per_row", "rejected_logps_per_row",+ )+ fail_on_nonfinite = True++ def _resolve_batch_size(self) -> int:+ """Never below 2, ``auto`` included (TRL refuses 1; see #1420)."""+ return max(KTO_MIN_BATCH, super()._resolve_batch_size())++ def _plan_batches(self, n_rows: int, batch_size: int, epochs: int, seed: int) -> list:+ """Dataset order, the last partial batch kept: TRL's sequential sampler.+ ``seed`` does not reorder anything, as on the transformers path."""+ return sequential_batches(n_rows, batch_size, epochs)++ def _check_train_split_size(self, dataset: dict) -> None:+ """Before the model loads: at least two rows (one KL group); fewer+ rows than ``batch_size`` train as one partial batch, as in TRL."""+ rows = len(dataset.get("train") or [])+ if 0 < rows < KTO_MIN_BATCH:+ self._check_train_rows(rows, self._resolve_batch_size(), 0)++ def _check_train_rows(self, n_rows: int, batch_size: int, ties: int) -> None:+ """Two rows or more: below ``batch_size`` they form one partial batch+ and one KL group, as TRL trains them."""+ if n_rows < KTO_MIN_BATCH:+ raise ValueError(+ f"MLX {self.objective}: the training split has {n_rows} row(s); KTO needs "+ f"at least {KTO_MIN_BATCH}, because a row's KL completion is another "+ "row's. Add rows."+ )++ def _on_plan_cut(self, dropped: list, kept: int, accum: int) -> None:+ """Batches run in dataset order, so the cut always drops the same rows."""+ rows = sorted({i for batch in dropped for i in batch})+ first, last = kept + 1, kept + len(dropped)+ span = f"{first}" if first == last else f"{first}-{last}"+ console.print(+ f"[yellow]MLX {self.objective}: gradient_accumulation_steps={accum} keeps "+ f"{kept} of {kept + len(dropped)} batches; batch(es) {span} of the run "+ f"({len(rows)} row(s): {', '.join(map(str, rows))}) are not trained. "+ "Rows run in dataset order, so these are the same rows every run; "+ "TRL instead updates on the partial window at each epoch's end.[/]"+ )++ def _examples(self, rows: list, split: str = "train") -> tuple[list, int]:+ """Every row, tokenized with its KL row; nothing is skipped, so the KL+ groups are TRL's. A bad row stops the run before the model trains."""+ if not rows:+ if split == "train":+ raise ValueError(f"MLX {self.objective}: the train split has no rows")+ return [], 0+ max_length = int(getattr(self.config.data, "max_length", 2048) or 2048)+ try:+ examples = build_kto_examples(+ rows, self.tokenizer, max_length=max_length,+ group_size=self._resolve_batch_size(),+ )+ except PreferenceRowError as exc:+ raise PreferenceRowError(f"{self.objective.lower()} {split} {exc}") from exc+ return examples, 0++ @staticmethod+ def _to_arrays(examples: list) -> tuple[Any, Any]:+ # DPO's reference precompute reads this: the matched rows' reference+ # log-probs come back as its first half, the KL rows' as its second.+ import mlx.core as mx++ tokens, mask = collate_kto(examples)+ return mx.array(tokens), mx.array(mask)++ def _batch_arrays(self, examples: list) -> tuple:+ import mlx.core as mx++ return (*self._to_arrays(examples), mx.array([bool(e.label) for e in examples]))++ def _loss(self, model: Any, arrays: tuple, split: str, rows: Any) -> tuple:+ tokens, mask, labels = arrays+ ref, ref_kl = (r[rows] for r in self._ref[split])+ policy, policy_kl = kto_logps(model, tokens, mask, ref.shape[0],+ head_split=self._head_split)+ return kto_loss(policy, policy_kl, ref, ref_kl, labels,+ float(self.config.training.kto_beta))++ def _format_metrics(self, metrics: dict) -> str:+ means = class_means(metrics)+ parts = [f"kl {metrics['kl']:.4f}"]+ for side in ("chosen", "rejected"):+ value = means.get(f"rewards_{side}")+ parts.append(f"rewards/{side} " + ("n/a" if value is None else f"{value:.4f}"))+ return ", ".join(parts)++ def _objective_metadata(self) -> dict:+ return {+ "task": "kto",+ "kto_beta": float(self.config.training.kto_beta),+ "kto_loss_type": KTO_LOSS_TYPE,+ "desirable_weight": DESIRABLE_WEIGHT,+ "undesirable_weight": UNDESIRABLE_WEIGHT,+ "reference": "precomputed from the base model",+ "kl_rows": "completions rotated by one within each kl_group_size rows",+ "kl_group_size": self._resolve_batch_size(),+ "batch_order": "dataset order (TRL's sequential sampler)",+ }++ def train(self, display=None, tracker=None, run_id=None, resume_from_checkpoint=None) -> dict:+ result = super().train(display=display, tracker=tracker, run_id=run_id,+ resume_from_checkpoint=resume_from_checkpoint)+ if "val_chosen_share" in result:+ result.update(class_means(result, prefix="val_"))+ return result++ def _load_checkpoint_weights(self, checkpoint_path: str) -> None:+ """Warm start only from a KTO adapter of the same base.++ The checkpoint's directory must hold the ``adapter_config.json`` an MLX+ KTO run writes (before its first step, so step checkpoints have it+ too). An adapter of another task, another base, or no metadata at all+ is refused before any tensor loads; MLX SFT's loader then checks the+ tensor names and shapes. The reference was already computed from the+ base model, before this runs.+ """+ import json+ from pathlib import Path++ name = self.objective+ meta_path = Path(checkpoint_path).resolve().parent / "adapter_config.json"+ if not meta_path.is_file():+ raise ValueError(+ f"MLX {name}: no adapter_config.json beside {checkpoint_path}; cannot "+ "confirm it is a task='kto' adapter of this base model"+ )+ try:+ meta = json.loads(meta_path.read_text(encoding="utf-8"))+ except (OSError, ValueError) as exc:+ raise ValueError(f"MLX {name}: unreadable {meta_path}: {exc}") from exc+ if meta.get("task") != "kto":+ raise ValueError(+ f"MLX {name}: {checkpoint_path} was trained with task="+ f"{meta.get('task')!r}, not 'kto'; start a fresh run or resume with "+ "the matching task"+ )+ if str(meta.get("model")) != str(self.config.base):+ raise ValueError(+ f"MLX {name}: {checkpoint_path} belongs to base {meta.get('model')!r}, "+ f"not {self.config.base!r}"+ )+ super()._load_checkpoint_weights(checkpoint_path)diff --git a/src/soup_cli/trainer/mlx_routing.py b/src/soup_cli/trainer/mlx_routing.pyindex 59696961..44fd6c5f 100644--- a/src/soup_cli/trainer/mlx_routing.py+++ b/src/soup_cli/trainer/mlx_routing.py@@ -28,6 +28,10 @@ class _MLXRegistry(dict):from soup_cli.trainer.mlx_simpo import MLXSimPOTrainerWrapperreturn MLXSimPOTrainerWrapper+ if task == "kto":+ from soup_cli.trainer.mlx_kto import MLXKTOTrainerWrapper++ return MLXKTOTrainerWrapperif task == "grpo":from soup_cli.trainer.mlx_grpo import MLXGRPOTrainerWrapper@@ -35,13 +39,13 @@ class _MLXRegistry(dict):raise KeyError(task)def __contains__(self, key: object) -> bool:- return key in ("sft", "dpo", "orpo", "simpo", "grpo")+ return key in ("sft", "dpo", "orpo", "simpo", "kto", "grpo")def __getitem__(self, key: str):return self._resolve(key)def keys(self): # noqa: D401- return ("sft", "dpo", "orpo", "simpo", "grpo")+ return ("sft", "dpo", "orpo", "simpo", "kto", "grpo")MLX_TRAINER_REGISTRY: _MLXRegistry = _MLXRegistry()diff --git a/tests/test_issue1213_dispatch_parity.py b/tests/test_issue1213_dispatch_parity.pyindex 77f16f39..d706ef12 100644--- a/tests/test_issue1213_dispatch_parity.py+++ b/tests/test_issue1213_dispatch_parity.py@@ -84,6 +84,7 @@ _MLX_EXPECTED = {"dpo": "MLXDPOTrainerWrapper","orpo": "MLXORPOTrainerWrapper","simpo": "MLXSimPOTrainerWrapper",+ "kto": "MLXKTOTrainerWrapper","grpo": "MLXGRPOTrainerWrapper",}diff --git a/tests/test_issue1324_mlx_loraplus_refusal.py b/tests/test_issue1324_mlx_loraplus_refusal.pyindex de3c86c8..6250171d 100644--- a/tests/test_issue1324_mlx_loraplus_refusal.py+++ b/tests/test_issue1324_mlx_loraplus_refusal.py@@ -94,7 +94,7 @@ def test_train_dry_run_exits_nonzero_on_mlx_loraplus(tmp_path: Path):@pytest.mark.parametrize("task, data_format",- [("ipo", "dpo"), ("kto", "kto"), ("pretrain", "plaintext"), ("embedding", "alpaca")],+ [("ipo", "dpo"), ("ppo", "dpo"), ("pretrain", "plaintext"), ("embedding", "alpaca")],)def test_a_task_mlx_cannot_run_is_refused_for_that_first(task, data_format):"""The MLX task gate runs first: switching backend is the whole fix there."""diff --git a/tests/test_issue1358_vocab_expansion_refused.py b/tests/test_issue1358_vocab_expansion_refused.pyindex 1ab20e12..f43dbc5d 100644--- a/tests/test_issue1358_vocab_expansion_refused.py+++ b/tests/test_issue1358_vocab_expansion_refused.py@@ -135,11 +135,13 @@ class TestTheOtherPathsThatDropThem:assert "data.add_new_tokens is not applied on backend='mlx'" in message, message@pytest.mark.parametrize("field", _FIELDS)- @pytest.mark.parametrize("task", ["dpo", "orpo", "simpo"])- def test_mlx_preference_loops(self, task, field):- """The native DPO, ORPO and SimPO loops tokenize with the base tokenizer as- it is; the refusal rests on each being in ``MLX_TASKS`` (review of #2073)."""- message = _refusal(_yaml(task, backend="mlx", field=field, data={"format": "dpo"}))+ @pytest.mark.parametrize("task, fmt", [+ ("dpo", "dpo"), ("orpo", "dpo"), ("simpo", "dpo"), ("kto", "kto"),+ ])+ def test_mlx_preference_loops(self, task, fmt, field):+ """The native DPO, ORPO, SimPO and KTO loops tokenize with the base tokenizer+ as it is; the refusal rests on each being in ``MLX_TASKS`` (review of #2073)."""+ message = _refusal(_yaml(task, backend="mlx", field=field, data={"format": fmt}))assert f"data.{field} is not applied on backend='mlx'" in message, message@@ -286,7 +288,6 @@ class TestABackendTheTaskCannotRunOnIsNamedFirst:@pytest.mark.parametrize(("task", "backend", "expected"), [("online_dpo", "unsloth", "task='online_dpo' requires backend='transformers'"),("ipo", "mlx", "MLX backend only ships SFT"),- ("kto", "mlx", "MLX backend only ships SFT"),])def test_the_backend_refusal_comes_first(self, task, backend, expected):message = _refusal(_yaml(task, backend=backend))diff --git a/tests/test_mlx_chat_template_refusal.py b/tests/test_mlx_chat_template_refusal.pyindex 5ac55198..94e593fb 100644--- a/tests/test_mlx_chat_template_refusal.py+++ b/tests/test_mlx_chat_template_refusal.py@@ -19,7 +19,7 @@ TEMPLATES = ["chatml", "llama3", "{% for m in messages %}{{ m['content'] }}{% enclass TestMlxChatTemplateRefusal:- @pytest.mark.parametrize("task", ["sft", "dpo", "orpo", "simpo"])+ @pytest.mark.parametrize("task", ["sft", "dpo", "orpo", "simpo", "kto"])@pytest.mark.parametrize("template", TEMPLATES)def test_mlx_with_chat_template_refused_at_config_load(self, template: str, task: str) -> None:with pytest.raises(ValueError) as exc_info:@@ -35,7 +35,7 @@ class TestMlxChatTemplateRefusal:assert "backend: transformers" in messageassert "Remove data.chat_template" in message- @pytest.mark.parametrize("task", ["sft", "dpo", "orpo", "simpo"])+ @pytest.mark.parametrize("task", ["sft", "dpo", "orpo", "simpo", "kto"])def test_empty_chat_template_still_loads_on_mlx(self, task: str) -> None:# The field validator normalises "" to None, which is not an override.config = SoupConfig(@@ -47,7 +47,7 @@ class TestMlxChatTemplateRefusal:)assert config.data.chat_template is None- @pytest.mark.parametrize("task", ["sft", "dpo", "orpo", "simpo"])+ @pytest.mark.parametrize("task", ["sft", "dpo", "orpo", "simpo", "kto"])@pytest.mark.parametrize("template", TEMPLATES)def test_transformers_backend_still_accepts_chat_template(self, template: str, task: str,@@ -62,7 +62,7 @@ class TestMlxChatTemplateRefusal:)assert config.data.chat_template == template- @pytest.mark.parametrize("task", ["sft", "dpo", "orpo", "simpo"])+ @pytest.mark.parametrize("task", ["sft", "dpo", "orpo", "simpo", "kto"])def test_mlx_without_chat_template_still_allowed(self, task: str) -> None:config = SoupConfig(base="test/model",diff --git a/tests/test_mlx_dpo.py b/tests/test_mlx_dpo.pyindex 26d43d84..f7774102 100644--- a/tests/test_mlx_dpo.py+++ b/tests/test_mlx_dpo.py@@ -468,9 +468,9 @@ class TestConfig:fields = {entry.field for entry in check_config(cfg)}assert fields == {"training.use_galore"}- # orpo and simpo have their own MLX trainers since #2004 / #1999- # (tests/test_mlx_orpo.py, tests/test_mlx_simpo.py).- @pytest.mark.parametrize("task", ["ipo", "kto", "ppo"])+ # orpo, simpo and kto have their own MLX trainers since #2004 / #1999 / #2000+ # (tests/test_mlx_orpo.py, tests/test_mlx_simpo.py, tests/test_mlx_kto.py).+ @pytest.mark.parametrize("task", ["ipo", "ppo"])def test_other_preference_tasks_still_refused(self, task):from soup_cli.config.loader import load_config_from_string@@ -716,6 +716,60 @@ class TestPromptCapIsWired:assert len(example.prompt_ids) + len(example.chosen_ids) <= 40+class TestThePlanHooksKeepDpoAndOrpoBehaviour:+ """Pins the 5ce31005 refactor (no MLX): DPO and ORPO still plan seeded,+ shuffled, whole batches and run ``auto`` at 1; only KTO overrides the hooks."""++ def _wrapper(self, cls, yaml_text):+ from soup_cli.config.loader import load_config_from_string++ return cls(load_config_from_string(yaml_text))++ def _both(self):+ from soup_cli.trainer.mlx_dpo import MLXDPOTrainerWrapper+ from soup_cli.trainer.mlx_orpo import MLXORPOTrainerWrapper+ from tests.test_mlx_orpo import _ORPO_YAML++ return [self._wrapper(MLXDPOTrainerWrapper, _MLX_DPO_YAML),+ self._wrapper(MLXORPOTrainerWrapper, _ORPO_YAML)]++ def test_the_plan_is_seeded_shuffled_and_drops_the_partial_batch(self):+ for wrapper in self._both():+ plan = wrapper._plan_batches(7, 2, 2, 3)+ assert plan == wrapper._plan_batches(7, 2, 2, 3)+ assert plan != wrapper._plan_batches(7, 2, 2, 4)+ assert len(plan) == 6 and all(len(b) == 2 for b in plan)+ for epoch in (plan[:3], plan[3:]): # 6 distinct rows; the 7th never trains+ rows = [i for b in epoch for i in b]+ assert len(set(rows)) == 6 and set(rows) <= set(range(7))+ sequential = [[0, 1], [2, 3], [4, 5]] * 2+ assert any(wrapper._plan_batches(7, 2, 2, s) != sequential for s in range(3))++ def test_auto_batch_size_is_one(self):+ for wrapper in self._both():+ wrapper.config.training.batch_size = "auto"+ assert wrapper._resolve_batch_size() == 1+ wrapper.config.training.batch_size = 3+ assert wrapper._resolve_batch_size() == 3++ def test_only_kto_overrides_the_hooks(self):+ from soup_cli.trainer.mlx_dpo import MLXDPOTrainerWrapper+ from soup_cli.trainer.mlx_kto import MLXKTOTrainerWrapper+ from soup_cli.trainer.mlx_orpo import MLXORPOTrainerWrapper++ for hook in ("_plan_batches", "_resolve_batch_size", "_check_train_rows",+ "_on_plan_cut"):+ base = getattr(MLXDPOTrainerWrapper, hook)+ assert getattr(MLXORPOTrainerWrapper, hook) is base, hook+ assert getattr(MLXKTOTrainerWrapper, hook) is not base, hook++ def test_fewer_pairs_than_a_batch_is_still_refused(self):+ wrapper = self._both()[0]+ with pytest.raises(ValueError, match=r"3 training pair\(s\) remain, fewer than "+ r"training.batch_size \(4\), after skipping 1"):+ wrapper._check_train_rows(3, 4, 1)++class TestConfigRefusals:def _load(self, extra_top="", extra_training=""):from soup_cli.config.loader import load_config_from_stringdiff --git a/tests/test_mlx_kto.py b/tests/test_mlx_kto.pynew file mode 100644index 00000000..661786fa--- /dev/null+++ b/tests/test_mlx_kto.py@@ -0,0 +1,1255 @@+"""Native KTO on MLX (#2000).++The objective is pinned to TRL 0.29.1's ``experimental/kto/kto_trainer.py``,+which Soup's transformers backend runs for ``task: kto`` (``trainer/kto.py``+forwards ``kto_beta`` and ``max_length`` only, so ``loss_type``,+``desirable_weight`` and ``undesirable_weight`` stay at ``KTOConfig``'s+defaults). Layers:++- the TRL facts the objective rests on, read from TRL itself (no MLX);+- tokenization and the KL (mismatched) rows against TRL's own ``_tokenize`` /+ ``_process_tokens`` / ``_get_kl_dataset`` pipeline on a real+ ``PreTrainedTokenizerFast`` (needs ``trl`` / ``datasets``, no MLX);+- an independent float64 NumPy oracle, tied to TRL's own ``kto_loss`` with+ torch (no MLX) and shown to be moved by every wrong objective the issue lists;+ the MLX loss, its terms and its gradients against that oracle (MLX);+- training, the fixed reference, saving and a fresh-process reload on a tiny+ random Llama (needs MLX and mlx-lm). No downloads.+"""++from __future__ import annotations++import ast+import inspect+import json+import math+import subprocess+import sys+import textwrap+import types+from pathlib import Path++import numpy as np+import pytest++from tests.test_mlx_dpo import _strip_ansi+from tests.test_mlx_orpo import (+ BOS,+ EOS,+ CharTokenizer,+ MergingTokenizer,+ _decode,+ _flat,+ _tiny_llama_args,+)++BETA = 0.1 # KTOConfig.beta's default and Soup's kto_beta default+++def _examples(rows, tokenizer=None, max_length=256, group_size=2):+ from soup_cli.trainer.mlx_kto import build_kto_examples++ return build_kto_examples(+ rows, tokenizer or CharTokenizer(), max_length=max_length, group_size=group_size,+ )+++def _one(row, **kw):+ return _examples([row, {"prompt": "z", "completion": "w", "label": False}], **kw)[0]+++# --------------------------------------------------------------------------+# TRL facts this objective rests on (no MLX)+# --------------------------------------------------------------------------+++def _kto_trainer_source() -> str:+ pytest.importorskip("trl")+ pytest.importorskip("torch")+ from trl.experimental.kto import kto_trainer++ return inspect.getsource(kto_trainer)+++class TestTrlFacts:+ def test_the_effective_settings_are_ktoconfig_defaults(self):+ pytest.importorskip("trl")+ from trl.experimental.kto import KTOConfig++ from soup_cli.trainer.mlx_kto import (+ DESIRABLE_WEIGHT,+ KTO_LOSS_TYPE,+ UNDESIRABLE_WEIGHT,+ )++ fields = KTOConfig.__dataclass_fields__+ assert fields["beta"].default == BETA+ assert fields["loss_type"].default == KTO_LOSS_TYPE == "kto"+ assert fields["desirable_weight"].default == DESIRABLE_WEIGHT == 1.0+ assert fields["undesirable_weight"].default == UNDESIRABLE_WEIGHT == 1.0+ assert fields["disable_dropout"].default is True+ # 0.29.1 has no prompt cap and no truncation mode: only max_length.+ assert not {"max_prompt_length", "max_completion_length", "truncation_mode"} & set(fields)++ def test_the_transformers_kto_path_passes_beta_and_max_length_only(self):+ """Soup builds KTOConfig with kto_beta and max_length; the prompt-length+ helper forwards nothing on 0.29.1, and no Soup cap touches the rows."""+ pytest.importorskip("trl")+ from trl.experimental.kto import KTOConfig++ import soup_cli.trainer.kto as kto+ from soup_cli.trainer._trl_compat import prompt_length_kwargs++ source = Path(kto.__file__).read_text(encoding="utf-8")+ calls = [+ node for node in ast.walk(ast.parse(source))+ if isinstance(node, ast.Call) and getattr(node.func, "id", "") == "KTOConfig"+ ]+ assert len(calls) == 1+ names = {kw.arg for kw in calls[0].keywords}+ assert {"beta", "max_length", "per_device_train_batch_size"} <= names+ assert not names & {"loss_type", "desirable_weight", "undesirable_weight",+ "precompute_ref_log_probs", "disable_dropout"}+ assert prompt_length_kwargs(KTOConfig, 64) == {}+ assert "enforce_preference_sequence_limit" not in source++ def test_the_training_sampler_is_sequential(self):+ """``_get_train_sampler`` (kto_trainer.py:1315-1320): TRL's KTO never+ shuffles, so a training batch is exactly one KL group."""+ torch = pytest.importorskip("torch")+ pytest.importorskip("trl")+ from trl.experimental.kto.kto_trainer import KTOTrainer++ sampler = KTOTrainer._get_train_sampler(types.SimpleNamespace(), list(range(5)))+ assert isinstance(sampler, torch.utils.data.SequentialSampler)++ def test_summed_logps_batch_floor_and_kl_grouping_are_in_the_source(self):+ source = _kto_trainer_source()+ assert source.count("average_log_prob=False") >= 4+ assert "if args.per_device_train_batch_size <= 1:" in source+ assert source.count("batch_size=args.per_device_train_batch_size,") == 2+ assert "kl = (policy_KL_logps - reference_KL_logps).mean().detach()" in source+ assert "kl = self.accelerator.gather_for_metrics(kl).mean().clamp(min=0)" in source+ assert "loss = losses.nanmean()" in source+++# --------------------------------------------------------------------------+# Tokenization (TRL 0.29.1 _tokenize / _process_tokens)+# --------------------------------------------------------------------------+++_U = {"role": "user", "content": "hi"}+++class TestTokenization:+ def test_a_generation_prompt_the_full_render_lacks_is_narrowed_away(self):+ """TRL keeps only the prefix the prompt and ``prompt + completion``+ renderings share (review of #2111): a template whose generation prompt+ adds text the full rendering does not contain must not shift the split."""+ from soup_cli.trainer.mlx_kto import render_kto_text++ class _Template:+ def apply_chat_template(self, messages, tokenize=False, add_generation_prompt=False,+ continue_final_message=False):+ text = "".join(f"{m['role'][0].upper()}:{m['content']}\n" for m in messages)+ return text + ("A:<think>\n" if add_generation_prompt else "")++ row = {"prompt": [{"role": "user", "content": "q"}],+ "completion": [{"role": "assistant", "content": "ans"}]}+ prompt, completion = render_kto_text(row, _Template())+ assert prompt == "U:q\nA:"+ assert completion == "ans\n"++ def test_text_row_gets_bos_on_the_prompt_and_eos_on_the_completion(self):+ ex = _one({"prompt": "Q:", "completion": " yes", "label": True})+ assert _decode(ex.prompt_ids) == "^Q:"+ assert _decode(ex.completion_ids) == " yes$"+ assert ex.label is True++ def test_bos_and_eos_are_not_doubled(self):+ class WithSpecials(CharTokenizer):+ def encode(self, text, add_special_tokens=True):+ ids = super().encode(text, False)+ return ([BOS] if text.startswith("Q") else []) + ids + (+ [EOS] if text.endswith("!") else [])++ ex = _one({"prompt": "Q", "completion": "a!", "label": True}, tokenizer=WithSpecials())+ whole = ex.prompt_ids + ex.completion_ids+ assert whole.count(BOS) == 1 and whole.count(EOS) == 1++ def test_conversational_row_renders_like_trl(self):+ ex = _one({"prompt": [_U], "completion": [{"role": "assistant", "content": "a"}],+ "label": False})+ assert _decode(ex.prompt_ids) == "^<user>hi|<assistant>"+ assert _decode(ex.completion_ids) == "a|$"++ def test_prompt_ending_in_assistant_continues_the_message(self):+ ex = _one({"prompt": [_U, {"role": "assistant", "content": "Sure,"}],+ "completion": [{"role": "assistant", "content": "a"}], "label": True})+ assert _decode(ex.prompt_ids) == "^<user>hi|<assistant>Sure,"++ def test_rows_go_through_the_loader_converter(self):+ """A string prompt beside a message completion is the user turn."""+ ex = _one({"prompt": "hi", "completion": [{"role": "assistant", "content": "a"}],+ "label": True})+ assert _decode(ex.prompt_ids) == "^<user>hi|<assistant>"++ def test_a_merged_last_prompt_token_moves_the_completion_start(self):+ ex = _one({"prompt": "zx", "completion": "y1", "label": True},+ tokenizer=MergingTokenizer())+ assert ex.completion_ids[0] == MergingTokenizer.MERGED+ assert _decode(ex.prompt_ids) == "^z"++ def test_an_overlong_row_cuts_the_completion_end_and_never_the_prompt(self):+ """TRL's rule: room is max_length minus the BOS / EOS it will add."""+ ex = _one({"prompt": "P" * 6, "completion": "abcdefgh", "label": True}, max_length=10)+ assert _decode(ex.prompt_ids) == "^" + "P" * 6+ assert _decode(ex.completion_ids) == "ab$"+ assert len(ex.prompt_ids) + len(ex.completion_ids) == 10+++class TestKLRows:+ def _rows(self, n):+ return [{"prompt": f"p{i}:", "completion": f" c{i}" + "x" * i, "label": i % 2 == 0}+ for i in range(n)]++ def test_partners_rotate_by_one_within_each_group(self):+ from soup_cli.trainer.mlx_kto import kl_partners++ assert kl_partners(7, 3) == [2, 0, 1, 5, 3, 4, 6]+ assert kl_partners(4, 2) == [1, 0, 3, 2]+ assert kl_partners(5, 2) == [1, 0, 3, 2, 4] # a last group of one is its own++ def test_each_kl_row_is_its_prompt_with_the_partner_completion(self):+ examples = _examples(self._rows(5), group_size=2)+ for i, j in enumerate([1, 0, 3, 2, 4]):+ assert examples[i].kl_completion_ids == examples[j].completion_ids+ assert _decode(examples[i].prompt_ids) == f"^p{i}:"++ def test_a_kl_completion_is_recut_to_its_own_prompt_room(self):+ rows = [{"prompt": "P" * 7, "completion": "ab", "label": True},+ {"prompt": "Q", "completion": "abcdefgh", "label": False}]+ a, b = _examples(rows, max_length=10, group_size=2)+ assert _decode(a.kl_completion_ids) == "a$" # room 1 after "^PPPPPPP"+ assert _decode(b.kl_completion_ids) == "ab$"++ def test_the_group_is_the_resolved_batch_size(self):+ from soup_cli.config.loader import load_config_from_string+ from soup_cli.trainer.mlx_kto import MLXKTOTrainerWrapper++ wrapper = MLXKTOTrainerWrapper(load_config_from_string(_yaml(batch_size=3)))+ wrapper.tokenizer = CharTokenizer()+ examples, skipped = wrapper._examples(self._rows(7))+ assert skipped == 0+ got = [next(j for j, e in enumerate(examples) if e.completion_ids == x.kl_completion_ids)+ for x in examples]+ assert got == [2, 0, 1, 5, 3, 4, 6]+++def _hf_tokenizer():+ pytest.importorskip("trl")+ pytest.importorskip("datasets")+ from tokenizers import Tokenizer, models, pre_tokenizers+ from transformers import PreTrainedTokenizerFast++ words = ["<s>", "</s>", "<pad>", "<unk>", "<", ">", "user", "assistant", "|", "Q", ":",+ "what", "is", "two", "plus", "one", "four", "five", "it", "yes", "no", "three",+ "the", "sum", "of", "six", "seven", "answer"]+ tok = Tokenizer(models.WordLevel(vocab={w: i for i, w in enumerate(words)},+ unk_token="<unk>"))+ tok.pre_tokenizer = pre_tokenizers.Sequence(+ [pre_tokenizers.Whitespace(), pre_tokenizers.Punctuation()]+ )+ fast = PreTrainedTokenizerFast(+ tokenizer_object=tok, bos_token="<s>", eos_token="</s>",+ pad_token="<pad>", unk_token="<unk>",+ )+ fast.chat_template = (+ "{% for m in messages %}<{{ m['role'] }}> {{ m['content'] }} | {% endfor %}"+ "{% if add_generation_prompt %}<assistant> {% endif %}"+ )+ return fast+++def _trl_pipeline(rows, tokenizer, max_length, batch_size):+ """TRL 0.29.1's own KTO dataset preparation, the order KTOTrainer.__init__ runs it."""+ import datasets+ from trl.data_utils import maybe_apply_chat_template+ from trl.experimental.kto.kto_trainer import _get_kl_dataset, _process_tokens, _tokenize++ ds = datasets.Dataset.from_list(rows)+ ds = ds.map(maybe_apply_chat_template, fn_kwargs={"tokenizer": tokenizer})+ ds = ds.map(_tokenize, batched=True, fn_kwargs={"tokenizer": tokenizer})+ kwargs = {"prefix": "", "tokenizer": tokenizer, "max_length": max_length}+ ds = ds.map(_process_tokens, fn_kwargs=kwargs)+ kl = ds.map(_get_kl_dataset, batched=True, batch_size=batch_size)+ kl = kl.map(_process_tokens, fn_kwargs={**kwargs, "prefix": "KL_"},+ remove_columns=[c for c in kl.column_names if c in ds.column_names])+ return datasets.concatenate_datasets([ds, kl], axis=1)+++_TEXT_ROWS = [+ {"prompt": "Q: what is two plus two", "completion": " four", "label": True},+ {"prompt": "Q: what is one plus one", "completion": " two", "label": False},+ {"prompt": "Q: is it five", "completion": " no it is four", "label": True},+ {"prompt": "Q: what is three", "completion": " three", "label": False},+ {"prompt": "Q: the sum of six", "completion": " six is the sum", "label": True},+ {"prompt": "Q: is seven one", "completion": " no", "label": False},+ {"prompt": "Q: what is the answer", "completion": " yes the answer is seven", "label": True},+]+++def _chat(rows):+ return [{"prompt": [{"role": "user", "content": r["prompt"][3:]}],+ "completion": [{"role": "assistant", "content": r["completion"].strip()}],+ "label": r["label"]} for r in rows]+++class TestTokensAndKLRowsMatchTrl:+ """Distinct prompts and answers, an uneven final group (7 rows, groups of 3)."""++ @pytest.mark.parametrize("rows, max_length, batch_size", [+ (_TEXT_ROWS, 64, 3),+ (_TEXT_ROWS, 10, 3), # several answers cut, and KL rows re-cut+ (_TEXT_ROWS, 64, 2),+ (_chat(_TEXT_ROWS), 64, 3),+ ])+ def test_ids_labels_and_kl_rows_match(self, rows, max_length, batch_size):+ tokenizer = _hf_tokenizer()+ trl = _trl_pipeline(rows, tokenizer, max_length, batch_size)+ ours = _examples(rows, tokenizer, max_length=max_length, group_size=batch_size)+ assert len(ours) == len(trl)+ for ex, ref in zip(ours, trl):+ assert ex.prompt_ids + ex.completion_ids == list(ref["completion_input_ids"])+ assert ex.prompt_ids + ex.kl_completion_ids == list(ref["KL_completion_input_ids"])+ for prefix in ("", "KL_"):+ labels = list(ref[f"{prefix}completion_labels"])+ assert sum(1 for x in labels if x == -100) == len(ex.prompt_ids)+ assert ex.label is ref["label"]+++# --------------------------------------------------------------------------+# The objective against a float64 NumPy oracle+# --------------------------------------------------------------------------+++def _sig(x):+ return 1.0 / (1.0 + np.exp(-x))+++def _np_kto(pol, pol_kl, ref, ref_kl, labels, beta=BETA, wd=1.0, wu=1.0, *, z=None,+ reverse=False, matched_z=False, clamp=True, equal_classes=False):+ """KTO as TRL 0.29.1 computes it; ``z`` frozen when given (it is detached)."""+ labels = np.asarray(labels, dtype=bool)+ if reverse:+ labels = ~labels+ if z is None:+ raw = (pol - ref).mean() if matched_z else (pol_kl - ref_kl).mean()+ z = max(raw, 0.0) if clamp else raw+ r = pol - ref+ losses = np.where(labels, wd * (1.0 - _sig(beta * (r - z))),+ wu * (1.0 - _sig(beta * (z - r))))+ if equal_classes:+ return 0.5 * (losses[labels].mean() + losses[~labels].mean())+ return losses.mean()+++def _np_z(pol_kl, ref_kl):+ return max((pol_kl - ref_kl).mean(), 0.0)+++#: Unbalanced labels (3 desirable, 1 undesirable); summed log-probs.+_POL = np.array([-6.2, -3.1, -8.4, -4.9])+_POL_KL = np.array([-7.0, -5.5, -6.1, -9.3])+_LABELS = np.array([True, False, True, True])+_DELTA = np.array([0.8, -0.5, 1.5, -1.2]) # r = policy - reference+++def _logps_case(kl_shift):+ """``kl_shift`` > 0 gives z = kl_shift; < 0 gives a negative raw z (clamped)."""+ return _POL, _POL_KL, _POL - _DELTA, _POL_KL - kl_shift, _LABELS+++#: (kl_shift, beta, desirable_weight, undesirable_weight)+_SETTINGS = [(0.7, 0.1, 1.0, 1.0), (-0.4, 0.1, 1.0, 1.0), (0.7, 0.5, 1.3, 0.7),+ (1.9, 1.0, 1.0, 2.0)]+++class TestTheOracleMatchesTrl:+ """No MLX: the oracle against TRL's own ``KTOTrainer.kto_loss`` (bound on a+ stub), so a CI runner without MLX still checks what the MLX tests compare to."""++ def _trl(self, pol, pol_kl, ref, ref_kl, labels, beta, wd, wu):+ torch = pytest.importorskip("torch")+ pytest.importorskip("trl")+ from trl.experimental.kto.kto_trainer import KTOTrainer++ stub = types.SimpleNamespace(+ calculate_KL=True, loss_type="kto", beta=beta, desirable_weight=wd,+ undesirable_weight=wu,+ accelerator=types.SimpleNamespace(gather_for_metrics=lambda x: x, device="cpu"),+ )+ lab = torch.tensor(labels)+ p = torch.tensor(pol, dtype=torch.float64, requires_grad=True)+ pk = torch.tensor(pol_kl, dtype=torch.float64, requires_grad=True)+ r = torch.tensor(ref, dtype=torch.float64)+ rk = torch.tensor(ref_kl, dtype=torch.float64)+ losses, chosen, rejected, kl = KTOTrainer.kto_loss(+ stub, p[lab], p[~lab], pk, r[lab], r[~lab], rk+ )+ loss = losses.nanmean()+ loss.backward()+ return loss.item(), kl.item(), p.grad.numpy(), pk.grad, chosen, rejected++ @pytest.mark.parametrize("shift, beta, wd, wu", _SETTINGS)+ def test_loss_z_rewards_and_gradient(self, shift, beta, wd, wu):+ pol, pol_kl, ref, ref_kl, labels = _logps_case(shift)+ loss, kl, grad, kl_grad, chosen, rejected = self._trl(+ pol, pol_kl, ref, ref_kl, labels, beta, wd, wu+ )+ assert _np_kto(pol, pol_kl, ref, ref_kl, labels, beta, wd, wu) == pytest.approx(+ loss, rel=1e-12)+ assert _np_z(pol_kl, ref_kl) == pytest.approx(kl, abs=1e-12)+ assert kl_grad is None # z is detached+ np.testing.assert_allclose(chosen.numpy(), beta * _DELTA[labels], rtol=1e-12)+ np.testing.assert_allclose(rejected.numpy(), beta * _DELTA[~labels], rtol=1e-12)+ eps = 1e-6+ z = _np_z(pol_kl, ref_kl)+ for i in range(len(pol)):+ hi, lo = pol.copy(), pol.copy()+ hi[i] += eps+ lo[i] -= eps+ fd = (_np_kto(hi, pol_kl, ref, ref_kl, labels, beta, wd, wu, z=z)+ - _np_kto(lo, pol_kl, ref, ref_kl, labels, beta, wd, wu, z=z)) / (2 * eps)+ assert fd == pytest.approx(grad[i], rel=1e-6, abs=1e-12), i++ def test_the_negative_case_really_clamps(self):+ _, pol_kl, _, ref_kl, _ = _logps_case(-0.4)+ assert (pol_kl - ref_kl).mean() == pytest.approx(-0.4)+ assert self._trl(*_logps_case(-0.4), BETA, 1.0, 1.0)[1] == 0.0++ @pytest.mark.parametrize("label", [True, False])+ def test_a_single_label_batch_is_the_mean_over_its_rows(self, label):+ pol, pol_kl, ref, ref_kl, _ = _logps_case(0.7)+ labels = np.array([label] * 4)+ loss = self._trl(pol, pol_kl, ref, ref_kl, labels, BETA, 1.0, 1.0)[0]+ assert math.isfinite(loss)+ assert _np_kto(pol, pol_kl, ref, ref_kl, labels) == pytest.approx(loss, rel=1e-12)+++class TestTheOracleSeesEveryListedMutation:+ @pytest.mark.parametrize("mutation", [+ "reverse", "matched_z", "no_clamp", "equal_classes",+ ])+ def test_each_mutation_moves_the_loss(self, mutation):+ case = _logps_case(-0.4 if mutation == "no_clamp" else 0.7)+ right = _np_kto(*case)+ flag = {"no_clamp": ("clamp", False)}.get(mutation, (mutation, True))+ wrong = _np_kto(*case, **dict([flag]))+ assert abs(wrong - right) > 1e-4++ def test_summed_and_mean_logps_differ_on_the_fixture(self):+ logits, tokens, mask, n = _fixture()+ assert not np.allclose(_np_logps(logits, tokens, mask),+ _np_logps(logits, tokens, mask, mean=True))+++def _np_logps(logits, tokens, mask, *, mean=False):+ shifted = logits[:, :-1] - logits[:, :-1].max(-1, keepdims=True)+ logsm = shifted - np.log(np.exp(shifted).sum(-1, keepdims=True))+ lp = np.take_along_axis(logsm, tokens[:, 1:][..., None], -1)[..., 0]+ m = mask[:, 1:].astype(np.float64)+ total = (lp * m).sum(-1)+ return total / m.sum(-1) if mean else total+++def _fixture(seed=0, vocab=12):+ """Four examples, three desirable, unequal prompt / completion / KL lengths."""+ from soup_cli.trainer.mlx_kto import KtoExample, collate_kto++ examples = [+ KtoExample([1, 5, 6], [7, 8, 2], [9, 10, 2], True),+ KtoExample([1, 3], [4, 2], [7, 8, 2], False),+ KtoExample([1, 11, 4, 6], [9, 10, 2], [4, 2], True),+ KtoExample([1, 8], [5, 5, 6, 2], [7, 2], True),+ ]+ tokens, mask = (np.array(x) for x in collate_kto(examples))+ rng = np.random.default_rng(seed)+ logits = rng.normal(0.0, 2.0, size=(tokens.shape[0], tokens.shape[1], vocab))+ return logits, tokens, mask, len(examples)+++class _RowKeyed:+ """Fixed logits per distinct token row, so separate forwards of the+ matched and KL halves each get their own rows' logits."""++ def __new__(cls, logits, tokens):+ nn = pytest.importorskip("mlx.nn")+ mx = pytest.importorskip("mlx.core")+ table = np.asarray(tokens)++ class Model(nn.Module):+ def __init__(self):+ super().__init__()+ self.logits = mx.array(logits.astype(np.float32))++ def __call__(self, x):+ rows = np.array(x)+ width = rows.shape[1]+ idx = [next(i for i, t in enumerate(table) if np.array_equal(t[:width], r))+ for r in rows]+ return self.logits[mx.array(idx), :width]++ return Model()+++def _refs(logits, tokens, mask, n, shift=0.7):+ pol = _np_logps(logits, tokens, mask)+ return pol[:n] - _DELTA, pol[n:] - shift+++class TestTheLossMatchesTheOracle:+ def _mlx(self, logits, tokens, mask, n, ref, ref_kl, beta=BETA, wd=1.0, wu=1.0):+ mx = pytest.importorskip("mlx.core")+ from soup_cli.trainer.mlx_kto import kto_logps, kto_loss++ model = _RowKeyed(logits, tokens)+ pol, pol_kl = kto_logps(model, mx.array(tokens), mx.array(mask), n)+ labels = mx.array(_LABELS)+ loss, metrics = kto_loss(pol, pol_kl, mx.array(ref), mx.array(ref_kl), labels, beta,+ desirable_weight=wd, undesirable_weight=wu)+ return np.array(pol), np.array(pol_kl), float(loss.item()), metrics++ @pytest.mark.parametrize("shift, beta, wd, wu", _SETTINGS)+ def test_terms_total_and_metrics(self, shift, beta, wd, wu):+ logits, tokens, mask, n = _fixture()+ ref, ref_kl = _refs(logits, tokens, mask, n, shift)+ pol, pol_kl, loss, metrics = self._mlx(logits, tokens, mask, n, ref, ref_kl,+ beta, wd, wu)+ want = _np_logps(logits, tokens, mask)+ np.testing.assert_allclose(pol, want[:n], rtol=1e-5)+ np.testing.assert_allclose(pol_kl, want[n:], rtol=1e-5)+ assert loss == pytest.approx(+ _np_kto(want[:n], want[n:], ref, ref_kl, _LABELS, beta, wd, wu), rel=1e-5)+ z = _np_z(want[n:], ref_kl)+ assert metrics["kl"].item() == pytest.approx(z, abs=1e-5)+ r = want[:n] - ref+ rows = len(_LABELS)+ assert metrics["chosen_share"].item() == pytest.approx(_LABELS.mean())+ assert metrics["chosen_rewards_per_row"].item() == pytest.approx(+ beta * r[_LABELS].sum() / rows, rel=1e-5)+ assert metrics["rejected_rewards_per_row"].item() == pytest.approx(+ beta * r[~_LABELS].sum() / rows, rel=1e-5)+ assert metrics["chosen_logps_per_row"].item() == pytest.approx(+ want[:n][_LABELS].sum() / rows, rel=1e-5)+ assert metrics["rejected_logps_per_row"].item() == pytest.approx(+ want[:n][~_LABELS].sum() / rows, rel=1e-5)++ def test_the_chunked_head_gives_the_same_logps(self):+ mx = pytest.importorskip("mlx.core")+ from soup_cli.trainer.mlx_kto import kto_logps++ logits, tokens, mask, n = _fixture()+ model = _RowKeyed(logits, tokens)+ split = (model, lambda h: h)+ full = kto_logps(model, mx.array(tokens), mx.array(mask), n)+ chunked = kto_logps(model, mx.array(tokens), mx.array(mask), n, head_split=split)+ for x, y in zip(full, chunked):+ assert mx.allclose(x, y, atol=1e-5).item()++ @pytest.mark.parametrize("shift, beta, wd, wu", [_SETTINGS[0], _SETTINGS[2]])+ def test_gradients_match_finite_differences_with_z_detached(self, shift, beta, wd, wu):+ mx = pytest.importorskip("mlx.core")+ nn = pytest.importorskip("mlx.nn")+ from soup_cli.trainer.mlx_kto import kto_logps, kto_loss++ logits, tokens, mask, n = _fixture(seed=1)+ ref, ref_kl = _refs(logits, tokens, mask, n, shift)+ model = _RowKeyed(logits, tokens)+ t, m = mx.array(tokens), mx.array(mask)++ def f(mod):+ pol, pol_kl = kto_logps(mod, t, m, n)+ return kto_loss(pol, pol_kl, mx.array(ref), mx.array(ref_kl), mx.array(_LABELS),+ beta, desirable_weight=wd, undesirable_weight=wu)[0]++ _, grads = nn.value_and_grad(model, f)(model)+ g = np.array(grads["logits"])+ # z is a constant: no gradient reaches the KL rows at all.+ assert np.abs(g[n:]).max() == 0.0+ assert np.abs(g[:n]).max() > 1e-4+ z = _np_z(_np_logps(logits, tokens, mask)[n:], ref_kl)++ def oracle(lg):+ pol = _np_logps(lg, tokens, mask)[:n]+ return _np_kto(pol, None, ref, None, _LABELS, beta, wd, wu, z=z)++ rng = np.random.default_rng(5)+ eps = 1e-4+ scored = np.argwhere(mask[:n, 1:] == 1)+ for _ in range(12):+ r, p = scored[rng.integers(0, len(scored))]+ idx = (r, p, rng.integers(0, logits.shape[2]))+ hi, lo = logits.copy(), logits.copy()+ hi[idx] += eps+ lo[idx] -= eps+ fd = (oracle(hi) - oracle(lo)) / (2 * eps)+ assert g[idx] == pytest.approx(fd, abs=2e-5, rel=2e-2), idx++ def test_prompt_and_padding_logits_move_nothing(self):+ logits, tokens, mask, n = _fixture()+ ref, ref_kl = _refs(logits, tokens, mask, n)+ base = self._mlx(logits, tokens, mask, n, ref, ref_kl)+ bumped = logits.copy()+ not_scored = mask[:, 1:] == 0+ bumped[:, :-1][not_scored] += np.random.default_rng(3).normal(+ 0.0, 3.0, size=(int(not_scored.sum()), logits.shape[-1]))+ moved = self._mlx(bumped, tokens, mask, n, ref, ref_kl)+ np.testing.assert_allclose(moved[0], base[0], rtol=1e-6)+ np.testing.assert_allclose(moved[1], base[1], rtol=1e-6)+ assert moved[2] == pytest.approx(base[2], rel=1e-6)+++class TestEdgeValues:+ def _loss(self, labels, shift=0.7):+ mx = pytest.importorskip("mlx.core")+ from soup_cli.trainer.mlx_kto import kto_loss++ pol, pol_kl, ref, ref_kl, _ = _logps_case(shift)+ return kto_loss(mx.array(pol), mx.array(pol_kl), mx.array(ref), mx.array(ref_kl),+ mx.array(np.asarray(labels)), BETA)++ @pytest.mark.parametrize("label", [True, False])+ def test_a_single_label_batch_is_finite_and_matches(self, label):+ labels = [label] * 4+ loss, metrics = self._loss(labels)+ pol, pol_kl, ref, ref_kl, _ = _logps_case(0.7)+ assert loss.item() == pytest.approx(_np_kto(pol, pol_kl, ref, ref_kl, labels), rel=1e-5)+ assert all(math.isfinite(v.item()) for v in metrics.values())++ def test_at_the_reference_the_loss_is_one_half_minus_the_kl_shift(self):+ """r = 0 and z = 0: every term is 1 - sigmoid(0) = 0.5."""+ mx = pytest.importorskip("mlx.core")+ from soup_cli.trainer.mlx_kto import kto_loss++ x = mx.array([-3.0, -4.0])+ loss, metrics = kto_loss(x, x, x, x, mx.array([True, False]), BETA)+ assert loss.item() == pytest.approx(0.5)+ assert metrics["kl"].item() == 0.0++ def test_nan_input_is_not_a_quiet_zero(self):+ mx = pytest.importorskip("mlx.core")+ from soup_cli.trainer.mlx_kto import kto_loss++ x = mx.array([-3.0, -4.0])+ loss, _ = kto_loss(x * mx.array([float("nan"), 1.0]), x, x, x,+ mx.array([True, False]), BETA)+ assert math.isnan(loss.item())+++# --------------------------------------------------------------------------+# Data refusals (no MLX)+# --------------------------------------------------------------------------+++class TestLabelsAreTheLoaders:+ def test_labels_read_through_load_dataset_match_the_transformers_path(+ self, tmp_path, monkeypatch+ ):+ """``load_dataset`` runs the loader's KTO converter for every backend: the+ transformers trainer trains on these rows' labels as they are, and MLX+ must read exactly the same ones (no second reading, no refusal)."""+ from soup_cli.config.schema import DataConfig+ from soup_cli.data.loader import load_dataset++ raw = [True, False, "true", 1, -1, "no", 0]+ path = tmp_path / "kto.jsonl"+ path.write_text("".join(+ json.dumps({"prompt": f"p{i}:", "completion": f" c{i}", "label": label}) + "\n"+ for i, label in enumerate(raw)+ ), encoding="utf-8")+ monkeypatch.chdir(tmp_path)+ loaded = load_dataset(DataConfig(train=str(path), format="kto", val_split=0.0))+ transformers_labels = [row["label"] for row in loaded["train"]]+ assert transformers_labels == [True, False, True, True, False, False, False]+ examples, _ = _wrapper_no_mlx()._examples(loaded["train"])+ assert [e.label for e in examples] == transformers_labels+++class TestDataRefusals:+ @pytest.mark.parametrize("label", ["maybe", None, 1, "true", 0.5])+ def test_a_row_bypassing_the_loader_with_a_non_bool_label_is_refused(self, label):+ """Only a guard for rows handed to the trainer without ``load_dataset``,+ which has already read every file row's label (test above)."""+ from soup_cli.trainer.mlx_dpo import PreferenceRowError++ rows = [{"prompt": "a", "completion": "b", "label": True},+ {"prompt": "c", "completion": "d", "label": label}]+ with pytest.raises(PreferenceRowError, match=r"kto train row 1: .*label must be"):+ _wrapper_no_mlx()._examples(rows)++ def test_a_missing_label_is_refused(self):+ from soup_cli.trainer.mlx_dpo import PreferenceRowError++ with pytest.raises(PreferenceRowError, match=r"kto val row 0: .*no 'label'"):+ _wrapper_no_mlx()._examples([{"prompt": "a", "completion": "b"}], "val")++ def test_an_empty_completion_is_refused(self):+ from soup_cli.trainer.mlx_dpo import PreferenceRowError++ rows = [{"prompt": "a", "completion": "", "label": True},+ {"prompt": "c", "completion": "d", "label": False}]+ with pytest.raises(PreferenceRowError, match="row 0: the completion is empty"):+ _wrapper_no_mlx()._examples(rows)++ @pytest.mark.parametrize("prompt_len, completion", [(10, "ok"), (16, "y" * 20)],+ ids=["fills-the-room", "longer-than-the-room"])+ def test_a_prompt_that_fills_max_length_is_refused(self, prompt_len, completion):+ """A prompt longer than the room must not reach TRL's negative slice,+ which keeps the head of a long answer past ``max_length`` (review of+ #2111): the second row would train with the clamp at 0 removed."""+ from soup_cli.trainer.mlx_dpo import PreferenceRowError++ wrapper = _wrapper_no_mlx()+ wrapper.config.data.max_length = 12+ rows = [{"prompt": "P" * prompt_len, "completion": completion, "label": True},+ {"prompt": "c", "completion": "d", "label": False}]+ with pytest.raises(PreferenceRowError, match="row 0: no completion token fits"):+ wrapper._examples(rows)++ def test_an_empty_training_split_is_refused(self):+ with pytest.raises(ValueError, match="MLX KTO: the train split has no rows"):+ _wrapper_no_mlx()._examples([])++ def test_fewer_rows_than_batch_size_form_one_partial_batch(self):+ """TRL trains 3 rows at batch_size 4 as one batch and one KL group."""+ from soup_cli.trainer.mlx_kto import kl_partners, sequential_batches++ wrapper = _wrapper_no_mlx()+ wrapper.config.training.batch_size = 4+ wrapper._check_train_split_size({"train": _rows(3)})+ wrapper._check_train_rows(3, 4, 0)+ assert sequential_batches(3, 4, 1) == [[0, 1, 2]]+ assert kl_partners(3, 4) == [2, 0, 1]++ @pytest.mark.parametrize("check", ["split", "rows"])+ def test_one_row_is_refused_in_kto_words(self, check):+ wrapper = _wrapper_no_mlx()+ with pytest.raises(ValueError) as info:+ if check == "split":+ wrapper._check_train_split_size({"train": _rows(1)})+ else:+ wrapper._check_train_rows(1, 2, 0)+ message = str(info.value)+ assert "MLX KTO: the training split has 1 row(s); KTO needs at least 2" in message+ assert "pair" not in message and "tie" not in message+ assert "Lower training.batch_size" not in message++ def test_an_empty_val_split_is_fine(self):+ assert _wrapper_no_mlx()._examples([], "val") == ([], 0)+++def _wrapper_no_mlx():+ from soup_cli.config.loader import load_config_from_string+ from soup_cli.trainer.mlx_kto import MLXKTOTrainerWrapper++ wrapper = MLXKTOTrainerWrapper(load_config_from_string(_KTO_YAML))+ wrapper.tokenizer = CharTokenizer()+ return wrapper+++class TestBatching:+ def test_batches_run_in_dataset_order_with_the_last_partial_one(self):+ """TRL's SequentialSampler and DataLoader(drop_last=False)."""+ from soup_cli.trainer.mlx_kto import sequential_batches++ assert sequential_batches(7, 2, 2) == [[0, 1], [2, 3], [4, 5], [6]] * 2+ assert sequential_batches(4, 2, 1) == [[0, 1], [2, 3]]++ def test_the_trainer_plans_sequential_batches(self):+ assert _wrapper_no_mlx()._plan_batches(5, 2, 1, 123) == [[0, 1], [2, 3], [4]]++ @pytest.mark.parametrize("value, expected", [("auto", 2), (2, 2), (5, 5)])+ def test_the_batch_size_never_drops_below_two(self, value, expected):+ wrapper = _wrapper_no_mlx()+ wrapper.config.training.batch_size = value+ assert wrapper._resolve_batch_size() == expected+++class TestClassMeans:+ """The reported ``val_rewards_*`` / ``val_logps_*`` from the per-row sums (no MLX)."""++ def test_each_class_mean_is_its_per_row_sum_over_its_share(self):+ from soup_cli.trainer.mlx_kto import class_means++ metrics = {+ "val_chosen_share": 0.25,+ "val_chosen_rewards_per_row": 0.5,+ "val_rejected_rewards_per_row": -0.6,+ "val_chosen_logps_per_row": -1.0,+ "val_rejected_logps_per_row": -6.0,+ }+ assert class_means(metrics, prefix="val_") == {+ "val_rewards_chosen": pytest.approx(2.0),+ "val_rewards_rejected": pytest.approx(-0.8),+ "val_logps_chosen": pytest.approx(-4.0),+ "val_logps_rejected": pytest.approx(-8.0),+ "val_rewards_margins": pytest.approx(2.8),+ }++ def test_a_split_with_one_label_reports_that_label_only(self):+ from soup_cli.trainer.mlx_kto import class_means++ metrics = {+ "chosen_share": 0.0,+ "chosen_rewards_per_row": 0.0,+ "rejected_rewards_per_row": -0.6,+ "chosen_logps_per_row": 0.0,+ "rejected_logps_per_row": -6.0,+ }+ assert class_means(metrics) == {+ "rewards_rejected": pytest.approx(-0.6),+ "logps_rejected": pytest.approx(-6.0),+ }+++# --------------------------------------------------------------------------+# Training on a tiny random Llama+# --------------------------------------------------------------------------++_KTO_YAML = """+base: tiny-base+task: kto+backend: mlx+data:+ train: ./x.jsonl+ format: kto+ max_length: 128+training:+ epochs: 1+ lr: 5e-3+ kto_beta: 0.1+ quantization: none+ lora: {r: 4, alpha: 8, dropout: 0.0}+output: ./output+"""+++def _rows(n=8, prefix="q"):+ return [{"prompt": f"{prefix}{i}:", "completion": " good" if i % 2 == 0 else " bad",+ "label": i % 2 == 0} for i in range(n)]+++def _yaml(**training):+ """The fixture with ``training`` keys replacing or adding lines."""+ text = _KTO_YAML+ for key, value in training.items():+ line = f" {key}: {value}\n"+ own = [x for x in text.splitlines(keepends=True) if x.startswith(f" {key}:")]+ text = text.replace(own[0], line) if own else text.replace(+ " kto_beta: 0.1\n", " kto_beta: 0.1\n" + line)+ return text+++def _wrapper(tmp_path, monkeypatch, model_seed=0, **training):+ mx = pytest.importorskip("mlx.core")+ pytest.importorskip("mlx_lm")+ from mlx_lm.models import llama++ from soup_cli.config.loader import load_config_from_string+ from soup_cli.trainer.mlx_kto import MLXKTOTrainerWrapper++ tmp_path.mkdir(parents=True, exist_ok=True)+ monkeypatch.chdir(tmp_path)+ training = {"batch_size": 2, "gradient_accumulation_steps": 1, **training}+ wrapper = MLXKTOTrainerWrapper(load_config_from_string(_yaml(**training)))+ mx.random.seed(model_seed)+ wrapper.model = llama.Model(_tiny_llama_args())+ wrapper.tokenizer = CharTokenizer()+ wrapper._dataset = {"train": _rows(), "val": _rows(2, prefix="v")}+ return wrapper+++def _base_logps(wrapper, examples, model_seed=0):+ """The untrained base model's summed log-probs: matched rows, then KL rows."""+ mx = pytest.importorskip("mlx.core")+ from mlx_lm.models import llama++ from soup_cli.trainer.mlx_dpo import sequence_logps++ mx.random.seed(model_seed)+ base = llama.Model(_tiny_llama_args())+ base.eval()+ tokens, mask = wrapper._to_arrays(examples)+ return np.array(sequence_logps(base, tokens, mask))+++class TestTraining:+ def test_lora_trains_and_the_base_stays_frozen(self, tmp_path, monkeypatch):+ wrapper = _wrapper(tmp_path, monkeypatch)+ before = _flat(wrapper.model.parameters())+ result = wrapper.train()+ assert result["losses"] and all(math.isfinite(x) for x in result["losses"])+ after = _flat(wrapper.model.parameters())+ lora_b = [v for k, v in after.items() if k.endswith("lora_b")]+ assert lora_b and any(np.abs(v).sum() > 0 for v in lora_b)+ for name, value in before.items():+ moved = name.replace(".weight", ".linear.weight")+ key = name if name in after else moved+ np.testing.assert_array_equal(after[key], value, err_msg=name)++ def test_the_reference_is_the_base_model_after_several_updates(self, tmp_path, monkeypatch):+ from soup_cli.trainer.mlx_dpo import sequence_logps++ wrapper = _wrapper(tmp_path, monkeypatch, epochs=3, warmup_ratio=0.0)+ result = wrapper.train()+ assert len(result["losses"]) >= 6+ examples, _ = wrapper._examples(_rows())+ base = _base_logps(wrapper, examples)+ ref_matched, ref_kl = (np.array(x) for x in wrapper._ref["train"])+ n = len(examples)+ np.testing.assert_allclose(ref_matched, base[:n], rtol=1e-4, atol=1e-4)+ np.testing.assert_allclose(ref_kl, base[n:], rtol=1e-4, atol=1e-4)+ wrapper.model.eval()+ trained = np.array(sequence_logps(wrapper.model, *wrapper._to_arrays(examples)))+ assert np.abs(trained - base).max() > 1e-2++ def test_the_preference_is_learned_and_the_metadata_written(self, tmp_path, monkeypatch):+ wrapper = _wrapper(tmp_path, monkeypatch, epochs=6, warmup_ratio=0.0)+ result = wrapper.train()+ assert all(math.isfinite(x) for x in result["losses"])+ assert result["val_rewards_chosen"] > 0 > result["val_rewards_rejected"]+ assert result["val_rewards_margins"] == pytest.approx(+ result["val_rewards_chosen"] - result["val_rewards_rejected"])+ assert result["val_kl"] >= 0+ meta = json.loads((tmp_path / "output" / "adapter_config.json").read_text())+ assert meta["task"] == "kto" and meta["fine_tune_type"] == "lora"+ assert (meta["kto_beta"], meta["kto_loss_type"]) == (0.1, "kto")+ assert (meta["desirable_weight"], meta["undesirable_weight"]) == (1.0, 1.0)+ assert meta["kl_group_size"] == 2+ assert not list((tmp_path / "output").glob("*.partial*"))++ def test_training_moves_held_out_rewards_against_the_untrained_model(+ self, tmp_path, monkeypatch+ ):+ """The base model is its own control: rewards are 0 before any update."""+ mx = pytest.importorskip("mlx.core")++ wrapper = _wrapper(tmp_path, monkeypatch, epochs=6, warmup_ratio=0.0)+ val = _rows(2, prefix="v")+ examples, _ = wrapper._examples(val, "val")+ control = _base_logps(wrapper, examples)+ wrapper.train()+ wrapper.model.eval()+ from soup_cli.trainer.mlx_dpo import sequence_logps++ tokens, mask = wrapper._to_arrays(examples)+ trained = np.array(sequence_logps(wrapper.model, tokens, mask))+ r = trained[:2] - control[:2]+ assert r[0] > 0.1 > -0.1 > r[1] # desirable up, undesirable down+ assert mx.array(r).shape == (2,)++ @pytest.mark.parametrize("label", [True, False])+ def test_single_label_data_trains_finitely(self, tmp_path, monkeypatch, label):+ wrapper = _wrapper(tmp_path, monkeypatch)+ wrapper._dataset["train"] = [dict(r, label=label) for r in _rows()]+ result = wrapper.train()+ assert all(math.isfinite(x) for x in result["losses"])++ def test_the_last_partial_batch_trains_and_val_chunks_weigh_by_rows(+ self, tmp_path, monkeypatch+ ):+ mx = pytest.importorskip("mlx.core")++ wrapper = _wrapper(tmp_path, monkeypatch)+ wrapper._dataset["train"] = _rows(7)+ wrapper._dataset["val"] = _rows(3, prefix="v")+ result = wrapper.train()+ assert result["total_steps"] == 4 # [0,1] [2,3] [4,5] [6], as TRL+ examples, _ = wrapper._examples(wrapper._dataset["val"], "val")+ wrapper.model.eval()+ chunks = [(0, 2), (2, 3)]+ total = 0.0+ for lo, hi in chunks:+ arrays = wrapper._batch_arrays(examples[lo:hi])+ loss, _ = wrapper._loss(wrapper.model, arrays, "val", mx.arange(lo, hi))+ total += loss.item() * (hi - lo)+ assert result["val_loss"] == pytest.approx(total / 3, rel=1e-4)++ def test_an_accumulated_update_is_the_mean_of_its_micro_batch_gradients(+ self, tmp_path, monkeypatch+ ):+ mx = pytest.importorskip("mlx.core")+ nn = pytest.importorskip("mlx.nn")+ from mlx.utils import tree_flatten++ settings = dict(gradient_accumulation_steps=2, optimizer="sgd", scheduler="constant",+ warmup_ratio=0.0, weight_decay=0.0, max_grad_norm=1000.0, seed=4,+ lr=0.5)+ trained = _wrapper(tmp_path / "a", monkeypatch, **settings)+ trained._dataset = {"train": _rows(4), "val": []}+ result = trained.train()+ assert len(result["losses"]) == 1+ got = _flat(trained.model.trainable_parameters())++ # The same start, by hand: seed, LoRA, reference, two micro-batch gradients.+ manual = _wrapper(tmp_path / "b", monkeypatch, **settings)+ mx.random.seed(4)+ examples, _ = manual._examples(_rows(4))+ manual._apply_lora(manual.model)+ manual._prepare_objective(examples, [], 2)+ start = _flat(manual.model.trainable_parameters())++ def micro(lo, hi):+ def f(model):+ return manual._loss(model, manual._batch_arrays(examples[lo:hi]), "train",+ mx.arange(lo, hi))[0]++ _, grads = nn.value_and_grad(manual.model, f)(manual.model)+ return {k: np.array(v) for k, v in tree_flatten(grads)}++ g1, g2 = micro(0, 2), micro(2, 4)+ assert any(np.abs(g1[k]).max() > 1e-5 for k in g1)+ for name, value in start.items():+ want = value - 0.5 * (g1[name] + g2[name]) / 2+ np.testing.assert_allclose(got[name], want, atol=2e-5, err_msg=name)++ def test_a_nonfinite_loss_stops_the_run(self, tmp_path, monkeypatch):+ mx = pytest.importorskip("mlx.core")+ from soup_cli.trainer import mlx_kto++ real = mlx_kto.kto_logps++ def poisoned(*args, **kwargs):+ pol, pol_kl = real(*args, **kwargs)+ return pol * mx.array(float("nan")), pol_kl++ monkeypatch.setattr(mlx_kto, "kto_logps", poisoned)+ wrapper = _wrapper(tmp_path, monkeypatch)+ with pytest.raises(FloatingPointError, match="MLX KTO: nonfinite loss"):+ wrapper.train()++ def test_a_bad_row_is_refused_before_any_update(self, tmp_path, monkeypatch):+ from soup_cli.trainer.mlx_dpo import PreferenceRowError++ wrapper = _wrapper(tmp_path, monkeypatch)+ wrapper._dataset["train"] = _rows(3) + [{"prompt": "x", "completion": "", "label": True}]+ with pytest.raises(PreferenceRowError, match="kto train row 3"):+ wrapper.train()+ assert not any("lora" in k for k in _flat(wrapper.model.parameters()))+ assert not (tmp_path / "output" / "adapters.safetensors").exists()+++class TestAccumulationCut:+ def test_the_warning_names_the_untrained_batches_and_rows(self, monkeypatch):+ from io import StringIO++ from rich.console import Console++ from soup_cli.trainer import mlx_kto++ buffer = StringIO()+ monkeypatch.setattr(mlx_kto, "console", Console(file=buffer, width=400))+ _wrapper_no_mlx()._on_plan_cut([[8, 9], [0, 1]], 8, 4)+ text = _strip_ansi(buffer.getvalue())+ assert "gradient_accumulation_steps=4 keeps 8 of 10 batches" in text+ assert "batch(es) 9-10 of the run (4 row(s): 0, 1, 8, 9) are not trained" in text++ def test_a_run_whose_plan_is_cut_warns_and_trains_the_kept_batches(+ self, tmp_path, monkeypatch+ ):+ from io import StringIO++ from rich.console import Console++ from soup_cli.trainer import mlx_kto++ buffer = StringIO()+ monkeypatch.setattr(mlx_kto, "console", Console(file=buffer, width=400))+ wrapper = _wrapper(tmp_path, monkeypatch, gradient_accumulation_steps=2)+ wrapper._dataset = {"train": _rows(5), "val": []} # [0,1] [2,3] [4]+ result = wrapper.train()+ assert result["total_steps"] == 2+ text = _strip_ansi(buffer.getvalue())+ assert "keeps 2 of 3 batches; batch(es) 3 of the run (1 row(s): 4)" in text++ def test_no_cut_no_warning(self, tmp_path, monkeypatch):+ from io import StringIO++ from rich.console import Console++ from soup_cli.trainer import mlx_kto++ buffer = StringIO()+ monkeypatch.setattr(mlx_kto, "console", Console(file=buffer, width=400))+ wrapper = _wrapper(tmp_path, monkeypatch, gradient_accumulation_steps=2)+ wrapper._dataset = {"train": _rows(4), "val": []}+ wrapper.train()+ assert "are not trained" not in _strip_ansi(buffer.getvalue())++ def test_three_rows_at_batch_four_train_as_one_batch(self, tmp_path, monkeypatch):+ wrapper = _wrapper(tmp_path, monkeypatch, batch_size=4)+ wrapper._dataset = {"train": _rows(3), "val": []}+ result = wrapper.train()+ assert result["total_steps"] == 1 and math.isfinite(result["losses"][0])+++class TestResume:+ def _first_run(self, tmp_path, monkeypatch):+ wrapper = _wrapper(tmp_path / "a", monkeypatch, epochs=3, seed=1)+ fresh = wrapper.train()+ return tmp_path / "a" / "output" / "adapters.safetensors", fresh++ def test_resume_from_its_own_adapter_keeps_the_base_reference(self, tmp_path, monkeypatch):+ checkpoint, fresh = self._first_run(tmp_path, monkeypatch)+ resumed = _wrapper(tmp_path / "b", monkeypatch, seed=1)+ result = resumed.train(resume_from_checkpoint=str(checkpoint))+ assert result["losses"][0] < fresh["losses"][0]+ examples, _ = resumed._examples(_rows())+ base = _base_logps(resumed, examples)+ np.testing.assert_allclose(np.array(resumed._ref["train"][0]), base[: len(examples)],+ rtol=1e-4, atol=1e-4)++ @pytest.mark.parametrize("change, match", [+ ({"task": "dpo"}, "MLX KTO: .* task='dpo', not 'kto'"),+ ({"task": "orpo"}, "task='orpo'"),+ ({"model": "other-base"}, "belongs to base"),+ (None, "MLX KTO: no adapter_config.json"),+ ])+ def test_mismatched_or_missing_metadata_is_refused(self, tmp_path, monkeypatch, change, match):+ checkpoint, _ = self._first_run(tmp_path, monkeypatch)+ meta = checkpoint.parent / "adapter_config.json"+ if change is None:+ meta.unlink()+ else:+ meta.write_text(json.dumps({**json.loads(meta.read_text()), **change}))+ resumed = _wrapper(tmp_path / "b", monkeypatch, seed=1)+ with pytest.raises(ValueError, match=match):+ resumed.train(resume_from_checkpoint=str(checkpoint))+++class TestFreshProcessReload:+ def test_logits_reproduce_and_the_adapter_has_an_effect(self, tmp_path, monkeypatch):+ mx = pytest.importorskip("mlx.core")++ wrapper = _wrapper(tmp_path, monkeypatch, epochs=3)+ base_path = tmp_path / "base.safetensors"+ wrapper.model.save_weights(str(base_path))+ wrapper.train()+ examples, _ = wrapper._examples(_rows(2, prefix="v"), "val")+ tokens, _ = wrapper._to_arrays(examples)+ wrapper.model.eval()+ logits = wrapper.model(tokens).astype(mx.float32)+ mx.eval(logits)+ np.save(tmp_path / "tokens.npy", np.array(tokens))++ script = textwrap.dedent(f"""+ import sys+ import numpy as np+ import mlx.core as mx+ from mlx_lm.models import llama+ from mlx_lm.tuner.utils import load_adapters+ args = llama.ModelArgs(model_type="llama", hidden_size=32, num_hidden_layers=2,+ intermediate_size=64, num_attention_heads=4, num_key_value_heads=2,+ rms_norm_eps=1e-5, vocab_size=200, tie_word_embeddings=True)+ model = llama.Model(args)+ model.load_weights({str(base_path)!r})+ if sys.argv[1] == "adapter":+ load_adapters(model, {str(tmp_path / "output")!r})+ model.eval()+ t = mx.array(np.load({str(tmp_path / "tokens.npy")!r}))+ np.save(sys.argv[2], np.array(model(t).astype(mx.float32)))+ """)++ def run(mode):+ out = tmp_path / f"{mode}.npy"+ subprocess.run([sys.executable, "-c", script, mode, str(out)], capture_output=True,+ text=True, check=True, timeout=300)+ return np.load(out)++ reloaded, base = run("adapter"), run("base")+ np.testing.assert_allclose(reloaded, np.array(logits), rtol=1e-4, atol=1e-5)+ assert np.abs(base - reloaded).max() > 1e-3+++# --------------------------------------------------------------------------+# Config and dispatch (no MLX)+# --------------------------------------------------------------------------+++class TestConfig:+ def _load(self, top="", training=""):+ """`training` lines replace the fixture's own line for the same key."""+ from soup_cli.config.loader import load_config_from_string++ text = _KTO_YAML.replace("backend: mlx\n", "backend: mlx\n" + top)+ for line in training.splitlines(keepends=True):+ key = line.split(":", 1)[0]+ own = [x for x in text.splitlines(keepends=True) if x.split(":", 1)[0] == key]+ text = text.replace(own[0], line) if own else text.replace(+ " kto_beta: 0.1\n", " kto_beta: 0.1\n" + line+ )+ return load_config_from_string(text)++ def test_kto_on_mlx_loads_and_is_an_mlx_task(self):+ from soup_cli.config.schema import MLX_TASKS++ cfg = self._load(training=" kto_beta: 0.3\n")+ assert (cfg.task, cfg.backend, cfg.training.kto_beta) == ("kto", "mlx", 0.3)+ assert "kto" in MLX_TASKS++ def test_pre_quantized_4bit_loads(self):+ assert self._load(training=" quantization: 4bit\n").training.quantization == "4bit"++ def test_kto_dispatches_to_the_native_trainer(self):+ from soup_cli.trainer.mlx_kto import MLXKTOTrainerWrapper+ from soup_cli.trainer.mlx_routing import MLX_TRAINER_REGISTRY, get_mlx_trainer++ assert "kto" in MLX_TRAINER_REGISTRY and "kto" in MLX_TRAINER_REGISTRY.keys()+ assert get_mlx_trainer("kto") is MLXKTOTrainerWrapper++ @pytest.mark.parametrize("top, training, match", [+ ("", " quantization: 8bit\n", "quantization='8bit' is not supported by task='kto'"),+ ("modality: vision\n", "", "modality='vision' is not supported by task='kto'"),+ ("", " lora: {r: 0}\n", "lora.r must be at least 1 for task='kto'"),+ ("", " loraplus_lr_ratio: 16\n", "loraplus_lr_ratio is not implemented on backend='mlx'"),+ ("", " lora: {r: 4, use_dora: true}\n", "MLX DoRA requires task='sft'"),+ ("", " batch_size: 1\n", "task='kto' requires training.batch_size >= 2"),+ ])+ def test_unsupported_combinations_are_refused_at_load(self, top, training, match):+ with pytest.raises(Exception, match=match):+ self._load(top, training)++ def test_batch_size_auto_loads(self):+ assert self._load(training=" batch_size: auto\n").training.batch_size == "auto"++ def test_other_tasks_name_kto_among_the_mlx_ones(self):+ from soup_cli.config.loader import load_config_from_string++ with pytest.raises(Exception, match="only ships SFT, DPO, ORPO, SimPO and KTO"):+ load_config_from_string(_KTO_YAML.replace("task: kto", "task: ipo"))++ def test_the_transformers_route_is_unchanged(self):+ from soup_cli.config.loader import load_config_from_string++ cfg = load_config_from_string(_KTO_YAML.replace("backend: mlx", "backend: transformers"))+ assert cfg.backend == "transformers" and cfg.task == "kto"++ def test_doctor_reports_ignored_settings(self):+ from soup_cli.config.backend_support import check_config++ cfg = self._load(training=" use_galore: true\n")+ assert {e.field for e in check_config(cfg)} == {"training.use_galore"}++ def test_the_ignored_settings_are_printed(self, monkeypatch):+ from io import StringIO++ from rich.console import Console++ from soup_cli.trainer import mlx_dpo+ from soup_cli.trainer.mlx_kto import MLXKTOTrainerWrapper++ buffer = StringIO()+ monkeypatch.setattr(mlx_dpo, "console", Console(file=buffer, width=400))+ MLXKTOTrainerWrapper(self._load(training=" use_galore: true\n"))._check_unsupported()+ assert "MLX backend ignores: GaLore" in _strip_ansi(buffer.getvalue())diff --git a/tests/test_mlx_lora_pattern_refusal.py b/tests/test_mlx_lora_pattern_refusal.pyindex 7158471f..6e221ac0 100644--- a/tests/test_mlx_lora_pattern_refusal.py+++ b/tests/test_mlx_lora_pattern_refusal.py@@ -26,7 +26,7 @@ from soup_cli.config.loader import load_config_from_stringfrom soup_cli.config.schema import SoupConfigfrom tests.conftest import strip_ansi-_MLX_TASKS = ["sft", "dpo", "orpo", "simpo"]+_MLX_TASKS = ["sft", "dpo", "orpo", "simpo", "kto"]_REMOVE_OR_SWITCH = "or use backend: transformers."_YAML = """@@ -181,7 +181,7 @@ class TestOtherRefusalsKeepTheirOwnWords:"""Each of these configs names a problem that "use backend: transformers" would notfix, or that has a more basic answer, so that refusal has to come first."""- @pytest.mark.parametrize("task", ["grpo", "kto", "ipo", "pretrain"])+ @pytest.mark.parametrize("task", ["grpo", "ipo", "pretrain"])def test_a_task_mlx_cannot_run_is_refused_for_that_first(self, task: str) -> None:with pytest.raises(ValidationError) as excinfo:_config(task, rank_pattern={"q_proj": 4})@@ -260,7 +260,7 @@ class TestOtherEntryPoints:flat = " ".join(text.split())assert ("`training.lora.rank_pattern` and `training.lora.alpha_pattern` are refused "- "on `backend: mlx` for SFT, DPO, ORPO and SimPO"+ "on `backend: mlx` for SFT, DPO, ORPO, SimPO and KTO") in flatdef test_the_doctor_table_has_no_pattern_rows(self) -> None:diff --git a/tests/test_mlx_simpo.py b/tests/test_mlx_simpo.pyindex 7f9fdd0f..06ec22fc 100644--- a/tests/test_mlx_simpo.py+++ b/tests/test_mlx_simpo.py@@ -527,6 +527,7 @@ class TestTraining:assert result["val_reward_accuracy"] == pytest.approx(1.0)assert result["val_reward_margin"] > 0assert math.isfinite(result["val_nll_loss"])+ assert math.isfinite(result["val_preference_loss"])assert result["skipped_identical_pairs"] == 0meta = json.loads((tmp_path / "output" / "adapter_config.json").read_text())assert meta["task"] == "simpo" and meta["fine_tune_type"] == "lora"@@ -755,7 +756,7 @@ class TestConfig:def test_other_tasks_name_simpo_among_the_mlx_ones(self):from soup_cli.config.loader import load_config_from_string- with pytest.raises(Exception, match="only ships SFT, DPO, ORPO and SimPO"):+ with pytest.raises(Exception, match="only ships SFT, DPO, ORPO, SimPO and KTO"):load_config_from_string(_SIMPO_YAML.replace("task: simpo", "task: ipo"))def test_the_transformers_route_is_unchanged(self):diff --git a/tests/test_v07204.py b/tests/test_v07204.pyindex e8ba59a2..e3dfa877 100644--- a/tests/test_v07204.py+++ b/tests/test_v07204.py@@ -1121,12 +1121,13 @@ class TestKtoBatchIsRefusedEarly:assert "KL term" in message and "gradient_accumulation_steps" in messageassert "streaming" not in message- def test_mlx_kto_reports_the_backend_refusal_first(self):+ def test_mlx_kto_keeps_the_same_floor(self):+ """MLX KTO (#2000) builds TRL's KL rows, so batch 1 is refused there too."""import yamlfrom soup_cli.config.loader import load_config_from_string- with pytest.raises(ValueError, match="MLX backend only ships SFT"):+ with pytest.raises(ValueError, match="task='kto' requires training.batch_size >= 2"):load_config_from_string(yaml.safe_dump({