Skip to content

armgit / Soup

Public Imported from GitHub
0

Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU.

feat(mlx): native KTO on Apple Silicon (#2000) (#2111)

dryn85-maxb815287f03b7
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index ac2efe11..3563c1f8 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -512,7 +512,7 @@ jobs:
# loss, gradient and training-loop tests ran on no job at all.
# tests/test_ci_mlx_smoke_files.py fails when a tests/test_mlx_*.py that
# needs the real library is neither listed here nor exempted there.
- run: pytest -o addopts= -q tests/test_rewind_mlx.py tests/test_mlx_dora_native.py tests/test_mlx_dpo.py tests/test_mlx_orpo.py tests/test_mlx_eval_steps.py tests/test_mlx_simpo.py tests/test_mlx_sft_seed.py
+ run: pytest -o addopts= -q tests/test_rewind_mlx.py tests/test_mlx_dora_native.py tests/test_mlx_dpo.py tests/test_mlx_orpo.py tests/test_mlx_eval_steps.py tests/test_mlx_simpo.py tests/test_mlx_sft_seed.py tests/test_mlx_kto.py
torchao-contract:
needs: plan
# Not on a pull request, approval run included: it is not a required check.
diff --git a/changelog.d/0.76.0/2111.added.md b/changelog.d/0.76.0/2111.added.md
new file mode 100644
index 00000000..614eb925
--- /dev/null
+++ b/changelog.d/0.76.0/2111.added.md
@@ -0,0 +1,12 @@
+- **KTO runs natively on Apple Silicon: `task: kto` with `backend: mlx` (#2000 by @dryn85-max in
+ #2111).** It was refused at config load. Soup's own MLX loop now trains TRL 0.29.1's KTO
+ objective with `training.kto_beta`: summed completion log-ratios against a reference
+ precomputed once from the base model, and a detached KL estimate built from TRL's own
+ mismatched rows. Batches run in dataset order, as in TRL. Rows that fit in `max_length` train
+ on the same tokens on either backend. Labels are read by the loader's KTO converter, exactly
+ as on the transformers backend. An empty completion, or a prompt that leaves no completion
+ room, is refused before training. `batch_size` stays at least 2, and a run whose gradient
+ accumulation cuts the last batches warns naming them. `--resume` refuses another task or
+ base, and a nonfinite loss stops the run. `quantization: 8bit`, `lora.r: 0`, non-text
+ modalities, DoRA and LoRA+ are refused on MLX. Try: `soup train --config soup.yaml` with
+ `task: kto`, `backend: mlx` and `format: kto` data.
diff --git a/docs/backends-and-ops.md b/docs/backends-and-ops.md
index 675f823f..a9d4d541 100644
--- a/docs/backends-and-ops.md
+++ b/docs/backends-and-ops.md
@@ -142,9 +142,9 @@ training:
alpha: 32
```
-MLX backend supports SFT, DPO, ORPO and SimPO. Any other task with `backend: mlx` (`grpo`, `ipo`, `kto`, `preference`, ...) is refused when the config is loaded, with an error naming the task. Requires `mlx-lm >= 0.31.3`. Use `soup recipes search mlx` for ready-made Apple Silicon configs.
+MLX backend supports SFT, DPO, ORPO, SimPO and KTO. Any other task with `backend: mlx` (`grpo`, `ipo`, `preference`, ...) is refused when the config is loaded, with an error naming the task. Requires `mlx-lm >= 0.31.3`. Use `soup recipes search mlx` for ready-made Apple Silicon configs.
-`training.lora.rank_pattern` and `training.lora.alpha_pattern` are refused on `backend: mlx` for SFT, DPO, ORPO and SimPO ([#1996](https://github.com/MakazhanAlpamys/Soup/issues/1996)): the MLX trainer builds every adapter from the one scalar `lora.r` and `lora.alpha`, so per-module ranks and alphas would be dropped without a word. Remove them, or use `backend: transformers`, which applies them. An empty `{}` sets nothing and still loads.
+`training.lora.rank_pattern` and `training.lora.alpha_pattern` are refused on `backend: mlx` for SFT, DPO, ORPO, SimPO and KTO ([#1996](https://github.com/MakazhanAlpamys/Soup/issues/1996)): the MLX trainer builds every adapter from the one scalar `lora.r` and `lora.alpha`, so per-module ranks and alphas would be dropped without a word. Remove them, or use `backend: transformers`, which applies them. An empty `{}` sets nothing and still loads.
#### DPO on MLX
@@ -286,11 +286,14 @@ That shows the objective moving, not alignment quality.
- **Stops on a nonfinite loss** instead of training on NaN.
- **LoRA dropout applies**, as in MLX DPO above; set `lora.dropout: 0` to
match TRL's `disable_dropout`.
-- **Reported:** loss, NLL, chosen and rejected rewards (`0.1 * logp`, as TRL
- logs them), reward accuracy and margin, and the mean chosen and rejected
- log-probabilities, per logged step and in the run result (`val_*` for the
- validation split). TRL's mean-logit metrics are not computed: the vocabulary
- head runs in chunks and the full logits are never held.
+- **Reported:** the panel and `soup ui` show loss and validation loss. Without a
+ live panel each logged step prints loss, NLL, reward accuracy and margin. The
+ run result has, for the validation split, `val_preference_loss` (the SimPO
+ term without the NLL, which is not on the per-step line), `val_nll_loss`, the
+ chosen and rejected rewards (`0.1 * logp`, as TRL logs them), reward accuracy
+ and margin, and the mean chosen and rejected log-probabilities. TRL's
+ mean-logit metrics are not computed: the vocabulary head runs in chunks and
+ the full logits are never held.
- **Saved:** `adapters.safetensors` and `adapter_config.json` (with `task:
simpo`, `simpo_beta`, `simpo_gamma`, `cpo_alpha` and the base model),
loadable with `mlx_lm.load(base, adapter_path=output)`, written as MLX ORPO
@@ -303,6 +306,112 @@ That shows the objective moving, not alignment quality.
`backend: mlx` (use `task: simpo`). The other settings MLX DPO does not apply
are not applied here either.
+#### KTO on MLX
+
+`task: kto` with `backend: mlx` runs the same loop and reference precompute as
+MLX DPO with KTO's objective, matched to TRL 0.29.1's KTO trainer (the one the
+transformers backend runs, with `training.kto_beta` as `beta` and every other
+`KTOConfig` setting at its default):
+
+- `r` is the policy minus the reference **summed** log-probability of the
+ completion;
+- `z = max(0, mean(policy_KL - reference_KL))` over the batch's KL rows,
+ without gradient;
+- a desirable row (`label: true`) loses `1 - sigmoid(kto_beta * (r - z))`, an
+ undesirable one `1 - sigmoid(kto_beta * (z - r))`, each weighted 1.0
+ (`desirable_weight` / `undesirable_weight` are not exposed), and the loss is
+ the mean over every row of the batch. A batch with only one label trains on
+ that label's rows alone.
+
+Qualified on Qwen2.5-0.5B-Instruct in bf16 and 4-bit on an M4 Pro: one epoch on
+1,200 unpaired UltraFeedback rows (`batch_size: 4`, accumulation 2) took about
+9 minutes and 8.0-9.3 GB, and the KTO loss on 400 held-out rows fell from 0.500
+to 0.448 (bf16) and 0.456 (4-bit). The desirable term fell while the undesirable
+one grew, the usual one-epoch KTO shape; that shows the objective moving, not
+alignment quality.
+
+```yaml
+base: mlx-community/Qwen2.5-0.5B-Instruct-4bit
+task: kto
+backend: mlx
+
+data:
+ train: ./kto.jsonl
+ format: kto
+ max_length: 1024
+
+training:
+ lr: 2e-5
+ kto_beta: 0.1
+ batch_size: 4
+ gradient_accumulation_steps: 4
+ gradient_checkpointing: true
+ lora:
+ r: 16
+ alpha: 32
+ dropout: 0
+```
+
+- **KL rows, as TRL builds them.** Each row's KL row is its own prompt with the
+ completion of the previous row in its group of `batch_size` consecutive rows,
+ the first row taking the last one's; a shorter last group rotates within
+ itself, and a last group of one row is its own KL row. Batches run in
+ dataset order, the last partial batch included, because TRL's KTO samples
+ sequentially: each training batch is one KL group, so its `z` is estimated
+ from the same completions it scores. With `gradient_accumulation_steps`
+ above 1 the plan is cut to whole accumulation windows, as in MLX DPO and
+ ORPO; in dataset order that always leaves the same final batches untrained
+ (TRL instead updates on the partial window at each epoch's end), and the run
+ warns before training, naming those batches and their rows. `training.seed` / `data_seed` do not
+ reorder rows (on either backend); shuffle the file to mix labels.
+- **Reference, precomputed.** The reference log-probs of every row and of its KL
+ row come from the base model before the first update, as in MLX DPO
+ (`--resume` included); the policy's KL log-probs are computed live, without
+ gradient, at every step.
+- **Tokens.** Rows are read by the `format: kto` converter and tokenized as
+ TRL's KTO trainer does: rendered to text with the chat template (keeping the
+ prefix the prompt and `prompt + completion` renderings share), the
+ completion located even when the tokenizer merges the last prompt token
+ into it, BOS prepended to the prompt and EOS appended to the completion when
+ missing. A row longer than `data.max_length` loses the end of its completion;
+ the prompt is never cut (TRL 0.29.1 has no prompt cap, and the transformers
+ path adds none). A KL row is re-cut to its own prompt's room.
+- **Labels** are read by the loader's `format: kto` converter, exactly as on the
+ transformers backend (`true` / `false`, and the strings and numbers that
+ converter accepts); MLX adds no reading of its own. Only a row handed to the
+ trainer without the loader is checked for a real `true` / `false`.
+- **Refused before any update:** a row with no `label`; an empty completion;
+ and a row whose prompt leaves no room for one completion token in
+ `data.max_length` (TRL keeps such a prompt whole and trains it past
+ `max_length`). A refusal names the split and row index.
+- **Batch size at least 2**, as on the transformers backend: `batch_size: 1` is
+ refused at load and `batch_size: auto` runs at 2. A training split smaller
+ than `batch_size` trains as one partial batch and one KL group, as in TRL;
+ fewer than 2 rows is refused, because a row's KL completion is another
+ row's.
+- **Stops on a nonfinite loss** instead of training on it (TRL's `nanmean`
+ drops a NaN row silently).
+- **LoRA dropout applies**, as in MLX DPO above; set `lora.dropout: 0` to match
+ TRL's `disable_dropout`. With dropout on, the KL forward draws its own mask.
+- **Reported:** the panel and `soup ui` show loss and validation loss. Without a
+ live panel each logged step also prints `kl`, `rewards/chosen` and
+ `rewards/rejected` (`kto_beta * r` averaged per label, as TRL logs them;
+ `n/a` for a label the step did not see). The run result has `val_kl`,
+ `val_rewards_chosen`, `val_rewards_rejected`, `val_rewards_margins`,
+ `val_logps_chosen` and `val_logps_rejected` for the validation split, which
+ is scored in `batch_size` chunks in order. TRL's `logits/*` and `count/*`
+ metrics are not reported.
+- **Saved:** `adapters.safetensors` and `adapter_config.json` (with `task: kto`,
+ `kto_beta`, the loss type, both weights, the KL group size and the base
+ model), loadable with `mlx_lm.load(base, adapter_path=output)`, written as
+ MLX ORPO writes them. `--resume` refuses a checkpoint without that metadata,
+ from another task, or from another base, before any tensor loads.
+- **Refused at load:** `quantization: 8bit` (use `none`, or a pre-quantized
+ 4-bit base with `quantization: 4bit`), `training.lora.r: 0` (the reference is
+ the untrained LoRA policy), non-text `modality`, DoRA, LoRA+ and
+ `data.chat_template` (as for MLX DPO and ORPO). The other settings MLX DPO
+ does not apply are not applied here either.
+
#### DoRA on MLX
For text SFT, set `training.lora.use_dora: true` to use MLX-LM's native
@@ -364,8 +473,8 @@ what ran is recoverable from the output directory.
### Transformers on MPS
-The regular `backend: transformers` path can run more than MLX's SFT, DPO, ORPO and SimPO
-surface. On a live MPS runtime that accepts bfloat16, Soup enables BF16 autocast
+The regular `backend: transformers` path can run more than MLX's SFT, DPO, ORPO,
+SimPO and KTO surface. On a live MPS runtime that accepts bfloat16, Soup enables BF16 autocast
for the hardware-validated text trainers: SFT, DPO, GRPO/RLVR, reward modelling,
and PRM.
The decision comes from a live MPS allocation probe rather than a macOS version
diff --git a/docs/models.md b/docs/models.md
index bc8956bb..0c480897 100644
--- a/docs/models.md
+++ b/docs/models.md
@@ -175,7 +175,7 @@ spelling is unchanged. See [the README's install section](../README.md#1-install
| `fast` | `pip install "soup-cli[fast]"` | Unsloth backend (2-5x faster, lower VRAM) |
| `vision` | `pip install "soup-cli[vision]"` | Vision / multimodal fine-tuning (Pillow) |
| `audio` | `pip install "soup-cli[audio]"` | Audio / speech fine-tuning (librosa, soundfile) |
-| `mlx` | `pip install "soup-cli[mlx]"` | Standalone Apple Silicon SFT, DPO, ORPO and SimPO backend for local data; `[train]` is not required |
+| `mlx` | `pip install "soup-cli[mlx]"` | Standalone Apple Silicon SFT, DPO, ORPO, SimPO and KTO backend for local data; `[train]` is not required |
| `qat` | `pip install "soup-cli[qat]"` | torchao: FP8 training (`quantization_aware: fp8`), FP8 attention, NVFP4, the `ao_adamw_*` optimizers and `soup export --format torchao` |
| `serve` | `pip install "soup-cli[serve]"` | Inference server (FastAPI + uvicorn) |
| `serve-fast` | `pip install "soup-cli[serve-fast]"` | vLLM inference backend (2-4x throughput) |
diff --git a/docs/training.md b/docs/training.md
index 887616a8..b9752740 100644
--- a/docs/training.md
+++ b/docs/training.md
@@ -1878,6 +1878,9 @@ training:
{"prompt": "What is 2+2?", "completion": "Fish", "label": false}
```
+On Apple Silicon, `backend: mlx` runs KTO natively with the same objective; see
+[KTO on MLX](backends-and-ops.md#kto-on-mlx).
+
## ORPO Training (No Reference Model)
diff --git a/src/soup_cli/config/backend_support.py b/src/soup_cli/config/backend_support.py
index 00196638..e4d75134 100644
--- a/src/soup_cli/config/backend_support.py
+++ b/src/soup_cli/config/backend_support.py
@@ -19,7 +19,7 @@ drift -- every entry must name a real field, and every dotted field name in
the truth; it can only stop it rotting, which is the failure that produced #749.
Scope today: ``task=sft`` on ``backend=mlx`` and on ``backend=unsloth`` (#1124;
-both apply to every modality), ``task=dpo``, ``task=orpo`` and ``task=simpo`` on
+both apply to every modality), ``task=dpo``, ``task=orpo``, ``task=simpo`` and ``task=kto`` on
``backend=mlx``, plus multimodal vision and audio on ``backend=transformers`` (#1156).
Other combinations report nothing rather than guessing, which is the honest default for
a table that has not been reviewed for them.
@@ -212,12 +212,16 @@ _MLX_ORPO: tuple[SupportEntry, ...] = _MLX_DPO
#: MLX SimPO is MLX ORPO's tokenization and MLX DPO's loop: the same fields.
_MLX_SIMPO: tuple[SupportEntry, ...] = _MLX_ORPO
+#: MLX KTO runs MLX DPO's loop and reference precompute: the same fields.
+_MLX_KTO: tuple[SupportEntry, ...] = _MLX_DPO
+
#: ``(task, backend, modality)`` -> the settings that combination does not honour.
REGISTRY: dict[tuple[str, str, str], tuple[SupportEntry, ...]] = {
("sft", "mlx", "text"): _MLX_SFT,
("dpo", "mlx", "text"): _MLX_DPO,
("orpo", "mlx", "text"): _MLX_ORPO,
("simpo", "mlx", "text"): _MLX_SIMPO,
+ ("kto", "mlx", "text"): _MLX_KTO,
("sft", "unsloth", "text"): _UNSLOTH_SFT,
("sft", "transformers", "vision"): _TRANSFORMERS_VISION_SFT,
("sft", "transformers", "audio"): _TRANSFORMERS_AUDIO_SFT,
@@ -271,6 +275,15 @@ TRAINER_MODULES: dict[tuple[str, str, str], tuple[str, ...]] = {
"soup_cli/trainer/_trl_compat.py",
"soup_cli/trainer/loss_summary.py",
),
+ ("kto", "mlx", "text"): (
+ "soup_cli/trainer/mlx_kto.py",
+ "soup_cli/trainer/mlx_orpo.py",
+ "soup_cli/trainer/mlx_dpo.py",
+ "soup_cli/trainer/mlx_sft.py",
+ "soup_cli/trainer/mlx_optim.py",
+ "soup_cli/trainer/_trl_compat.py",
+ "soup_cli/trainer/loss_summary.py",
+ ),
# utils/unsloth.py handles the loader path. trainer/sft.py::_setup_unsloth is not listed
# here because listing sft.py would cause the drift guard to inspect transformers-only
# precision reads; an AST test in test_issue1124 guards _setup_unsloth instead.
diff --git a/src/soup_cli/config/schema.py b/src/soup_cli/config/schema.py
index 31a30044..f25846ae 100644
--- a/src/soup_cli/config/schema.py
+++ b/src/soup_cli/config/schema.py
@@ -5109,9 +5109,10 @@ UNSLOTH_SETUP_TASKS: frozenset[str] = frozenset({
})
#: The tasks with a trainer on ``backend: mlx``: SFT through mlx-lm's tuner and
-#: DPO, ORPO and SimPO through Soup's own loop (``trainer/mlx_dpo.py``). Validators that defer
-#: to :meth:`SoupConfig._validate_mlx_task_support` read this, not ``"sft"``.
-MLX_TASKS: frozenset[str] = frozenset({"sft", "dpo", "orpo", "simpo"})
+#: DPO, ORPO, SimPO and KTO through Soup's own loop (``trainer/mlx_dpo.py``).
+#: Validators that defer to :meth:`SoupConfig._validate_mlx_task_support` read
+#: this, not ``"sft"``.
+MLX_TASKS: frozenset[str] = frozenset({"sft", "dpo", "orpo", "simpo", "kto"})
#: #1497 — the tasks whose trainer reads ``training.freeze_layers`` /
#: ``training.freeze_ratio``: ``SFTTrainerWrapper._setup_transformers``, which
@@ -7036,8 +7037,9 @@ class SoupConfig(BaseModel):
I/O on a real base. The same refusal now applies to every
``task='kto'`` config, resident or streamed, before anything loads.
"""
- if self.backend == "mlx":
+ if self.backend == "mlx" and self.task not in MLX_TASKS:
return self # _validate_mlx_task_support gives the more basic answer
+ # MLX KTO (#2000) builds TRL's KL rows and keeps the same floor.
tcfg = self.training
if (
self.task == "kto"
@@ -8581,23 +8583,23 @@ class SoupConfig(BaseModel):
@model_validator(mode="after")
def _validate_mlx_task_support(self) -> "SoupConfig":
- """MLX backend supports sft, dpo, orpo and simpo.
+ """MLX backend supports sft, dpo, orpo, simpo and kto.
- SFT runs through mlx-lm's tuner; DPO, ORPO and SimPO through Soup's own
- loop in ``trainer/mlx_dpo.py`` / ``trainer/mlx_orpo.py`` /
- ``trainer/mlx_simpo.py``. Every other task -- GRPO included, whose MLX
- wrapper is still scaffolding -- is refused here rather than at
- ``train()`` time.
+ SFT runs through mlx-lm's tuner; DPO, ORPO, SimPO and KTO through Soup's
+ own loop in ``trainer/mlx_dpo.py`` / ``trainer/mlx_orpo.py`` /
+ ``trainer/mlx_simpo.py`` / ``trainer/mlx_kto.py``. Every other task --
+ GRPO included, whose MLX wrapper is still scaffolding -- is refused here
+ rather than at ``train()`` time.
"""
if self.backend != "mlx":
return self
- if self.task in ("dpo", "orpo", "simpo"):
+ if self.task in ("dpo", "orpo", "simpo", "kto"):
# The native preference loops tokenize text and chat rows only and
- # train a LoRA adapter. DPO's reference log-probs come from the
- # policy before training, which is the base model only because a
- # LoRA adapter's B starts at zero; ORPO and SimPO train LoRA only
- # (no full fine-tune on MLX yet).
- name = {"dpo": "DPO", "orpo": "ORPO", "simpo": "SimPO"}[self.task]
+ # train a LoRA adapter. DPO's and KTO's reference log-probs come
+ # from the policy before training, which is the base model only
+ # because a LoRA adapter's B starts at zero; ORPO and SimPO train
+ # LoRA only (no full fine-tune on MLX yet).
+ name = {"dpo": "DPO", "orpo": "ORPO", "simpo": "SimPO", "kto": "KTO"}[self.task]
if self.modality != "text":
raise ValueError(
f"modality='{self.modality}' is not supported by task='{self.task}' "
@@ -8608,7 +8610,7 @@ class SoupConfig(BaseModel):
why = (
"the reference model is the untrained LoRA policy, which needs "
"a LoRA adapter to exist"
- if self.task == "dpo"
+ if self.task in ("dpo", "kto")
else f"MLX {name} trains a LoRA adapter; full fine-tuning is not "
"implemented on backend='mlx'"
)
@@ -8616,7 +8618,7 @@ class SoupConfig(BaseModel):
f"training.lora.r must be at least 1 for task='{self.task}' on "
f"backend='mlx': {why}."
)
- if self.task in ("orpo", "simpo") and self.training.quantization == "8bit":
+ if self.task in ("orpo", "simpo", "kto") and self.training.quantization == "8bit":
raise ValueError(
f"training.quantization='8bit' is not supported by task='{self.task}' "
"on backend='mlx': use quantization: none, or a pre-quantized "
@@ -8630,8 +8632,8 @@ class SoupConfig(BaseModel):
if self.task == "preference" and self.training.preference_loss in native:
hint = f" Use task: {self.training.preference_loss} for the native MLX trainer."
raise ValueError(
- f"MLX backend only ships SFT, DPO, ORPO and SimPO; task='{self.task}' has no "
- "trainer on backend='mlx'."
+ f"MLX backend only ships SFT, DPO, ORPO, SimPO and KTO; task='{self.task}' has "
+ "no trainer on backend='mlx'."
+ (hint or f" Use backend=transformers for task={self.task}.")
)
@@ -8667,9 +8669,10 @@ class SoupConfig(BaseModel):
``backend: mlx``. Stop-gap: delete this validator (and
``tests/test_mlx_lora_pattern_refusal.py``) when the MLX trainer honours them.
- SFT, DPO, ORPO and SimPO all build the adapter in ``MLXSFTTrainerWrapper._apply_lora``
- from one scalar ``lora.r`` and ``lora.alpha``, so a config that set a pattern
- trained a uniform-rank adapter without a word. Not part of
+ SFT, DPO, ORPO, SimPO and KTO all build the adapter in
+ ``MLXSFTTrainerWrapper._apply_lora`` from one scalar ``lora.r`` and
+ ``lora.alpha``, so a config that set a pattern trained a uniform-rank adapter
+ without a word. Not part of
:meth:`_validate_peft_variant_backend_and_quantization`, which returns early
for plain LoRA; defined after :meth:`_validate_mlx_task_support` (and the
``lora.r=0`` check) on purpose, so a task MLX cannot run, or a config with no
diff --git a/src/soup_cli/trainer/mlx_dpo.py b/src/soup_cli/trainer/mlx_dpo.py
index 60a811af..e3ca1ee1 100644
--- a/src/soup_cli/trainer/mlx_dpo.py
+++ b/src/soup_cli/trainer/mlx_dpo.py
@@ -427,7 +427,10 @@ class MLXDPOTrainerWrapper(MLXSFTTrainerWrapper):
self.model.eval()
chosen, rejected = [], []
total = len(examples)
- console.print(f"[dim]MLX DPO: computing reference log-probs for {total} pair(s)[/]")
+ console.print(
+ f"[dim]MLX {self.objective}: computing reference log-probs for {total} "
+ f"{self.reference_unit}[/]"
+ )
every = max(batch_size, (total // 10) // batch_size * batch_size)
for start in range(0, total, batch_size):
if start and start % every == 0:
@@ -510,6 +513,33 @@ class MLXDPOTrainerWrapper(MLXSFTTrainerWrapper):
"""``(tokens, completion mask)`` for the head-split check."""
return self._to_arrays(examples)
+ #: What one precomputed reference entry scores, for the progress message.
+ reference_unit = "pair(s)"
+
+ def _resolve_batch_size(self) -> int:
+ """``training.batch_size`` as this loop runs it (``auto`` is 1 here)."""
+ tcfg = self.config.training
+ return int(tcfg.batch_size) if isinstance(tcfg.batch_size, int) else 1
+
+ def _plan_batches(self, n_rows: int, batch_size: int, epochs: int, seed: int) -> list:
+ """Every epoch's batches of example indices, in training order."""
+ return plan_batches(n_rows, batch_size, epochs, seed)
+
+ def _check_train_rows(self, n_rows: int, batch_size: int, ties: int) -> None:
+ """Refuse a training split this objective cannot batch."""
+ if n_rows < batch_size:
+ raise ValueError(
+ f"MLX {self.objective}: {n_rows} training pair(s) remain, fewer than "
+ f"training.batch_size ({batch_size})"
+ + (f", after skipping {ties} tie(s)" if ties else "")
+ + f". Lower training.batch_size to at most {max(1, n_rows)} "
+ "or add rows."
+ )
+
+ def _on_plan_cut(self, dropped: list, kept: int, accum: int) -> None:
+ """Called with the batches an accumulation cut leaves untrained; the
+ shuffled plans of DPO and ORPO drop different rows each run, silently."""
+
def _check_finite(self, loss: Any, split: str) -> None:
import math as _math
@@ -576,7 +606,7 @@ class MLXDPOTrainerWrapper(MLXSFTTrainerWrapper):
output_dir = Path(cfg.output)
output_dir.mkdir(parents=True, exist_ok=True)
- batch_size = int(tcfg.batch_size) if isinstance(tcfg.batch_size, int) else 1
+ batch_size = self._resolve_batch_size()
accum = max(1, int(tcfg.gradient_accumulation_steps))
train_examples, train_ties = self._examples(list(self._dataset.get("train", [])))
@@ -587,22 +617,19 @@ class MLXDPOTrainerWrapper(MLXSFTTrainerWrapper):
f"[yellow]MLX {name}: skipped {skipped} {name.lower()} row(s) with "
"identical chosen and rejected (a tie carries no preference signal)[/]"
)
- if len(train_examples) < batch_size:
- raise ValueError(
- f"MLX {name}: {len(train_examples)} training pair(s) remain, fewer than "
- f"training.batch_size ({batch_size})"
- + (f", after skipping {train_ties} tie(s)" if train_ties else "")
- + f". Lower training.batch_size to at most {max(1, len(train_examples))} "
- "or add rows."
- )
- plan = plan_batches(len(train_examples), batch_size, int(tcfg.epochs), data_seed)
+ self._check_train_rows(len(train_examples), batch_size, train_ties)
+ epoch_count = max(1, int(tcfg.epochs))
+ plan = self._plan_batches(len(train_examples), batch_size, epoch_count, data_seed)
+ batches_per_epoch = max(1, len(plan) // epoch_count)
iters = len(plan)
if accum > 1:
# Like MLX SFT: no partial accumulation group is ever flushed.
- iters = max(accum, iters - iters % accum)
+ cut = max(accum, iters - iters % accum)
+ if cut < iters:
+ self._on_plan_cut(plan[cut:], cut, accum)
+ iters = cut
plan = (plan * math.ceil(iters / len(plan)))[:iters]
total_updates = max(1, iters // accum)
- batches_per_epoch = max(1, len(train_examples) // batch_size)
self._apply_lora(self.model)
if tcfg.gradient_checkpointing:
diff --git a/src/soup_cli/trainer/mlx_kto.py b/src/soup_cli/trainer/mlx_kto.py
new file mode 100644
index 00000000..c4372ab1
--- /dev/null
+++ b/src/soup_cli/trainer/mlx_kto.py
@@ -0,0 +1,479 @@
+"""MLX KTO trainer — unpaired preference alignment on Apple Silicon.
+
+The objective is TRL 0.29.1's ``experimental/kto/kto_trainer.py``, the trainer
+Soup's transformers backend runs for ``task: kto`` (``trainer/kto.py`` forwards
+``kto_beta`` and ``max_length``; everything else stays at ``KTOConfig``'s
+defaults: ``loss_type="kto"``, both class weights 1.0):
+
+- ``r`` = policy minus reference **summed** completion log-probability
+ (``get_batch_logps(..., average_log_prob=False)``);
+- ``z`` = ``mean(policy_KL - reference_KL)`` over the batch's KL rows,
+ detached, then clamped at 0 (``kto_loss``);
+- desirable rows lose ``w_d * (1 - sigmoid(beta * (r - z)))``, undesirable rows
+ ``w_u * (1 - sigmoid(beta * (z - r)))``; the loss is the mean over every row
+ of the batch (TRL concatenates both classes, then takes ``nanmean``), so a
+ class missing from a batch simply contributes nothing.
+
+The KL rows are TRL's: each example's prompt with the completion of the
+previous example in its group, the first taking the last (``_get_kl_dataset``,
+mapped with ``batch_size=per_device_train_batch_size``; a last, shorter group
+rotates within itself, and a group of one is its own KL row). TRL's KTO samples
+batches **sequentially** (``_get_train_sampler`` returns a ``SequentialSampler``)
+and keeps the last partial batch, so a training batch is exactly one KL group;
+this trainer plans its batches the same way.
+
+The loop, LoRA attach, optimizer plan, clipping, seeding, chunked head, adapter
+writer and reference precompute are MLX DPO's
+(:class:`~soup_cli.trainer.mlx_dpo.MLXDPOTrainerWrapper`): the reference
+log-probs of every matched and KL row are computed once from the base model
+before the first update (LoRA ``B`` starts at zero), and the policy's KL
+log-probs are computed live, without gradient.
+
+Tokenization is TRL's KTO ``_tokenize`` / ``_process_tokens``: rows are rendered
+to text first (``apply_chat_template`` with TRL's common-prefix rule), the
+completion is located with TRL's one-token merge rule, BOS is prepended to the
+prompt and EOS appended to the completion when missing, and an over-long row
+loses the end of its completion (the prompt is never cut; there is no prompt
+cap in 0.29.1). A row whose completion keeps no token is refused, where TRL
+would train the prompt plus a bare EOS (or slice past the prompt).
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any
+
+from rich.console import Console
+
+from soup_cli.trainer.mlx_dpo import (
+ MLXDPOTrainerWrapper,
+ PreferenceRowError,
+ sequence_logps,
+)
+from soup_cli.trainer.mlx_orpo import _common_prefix, _tokenized_answer
+
+console = Console()
+
+#: ``KTOConfig``'s defaults in TRL 0.29.1, which Soup's transformers path keeps.
+KTO_LOSS_TYPE = "kto"
+DESIRABLE_WEIGHT = 1.0
+UNDESIRABLE_WEIGHT = 1.0
+
+#: TRL refuses a per-device batch of 1: the KL row would be the matched row.
+KTO_MIN_BATCH = 2
+
+
+@dataclass(frozen=True)
+class KtoExample:
+ """One tokenized row: its prompt, its completion and its KL completion.
+
+ The KL row shares the prompt; its completion is another row's, re-cut to
+ this prompt's room, as TRL's second ``_process_tokens`` pass does.
+ """
+
+ prompt_ids: list
+ completion_ids: list
+ kl_completion_ids: list
+ label: bool
+
+
+# --------------------------------------------------------------------------
+# Tokenization (TRL 0.29.1 KTO _tokenize / _process_tokens / _get_kl_dataset)
+# --------------------------------------------------------------------------
+
+
+def _normalise_row(row: dict) -> dict:
+ """The row in one layout via the loader's converter.
+
+ Labels are read by the loader's KTO converter (``_convert_kto``) when
+ ``load_dataset`` loads the file, exactly as on the transformers backend, so
+ every row that reaches the trainer that way already has a bool label. The
+ bool check below only guards rows handed to ``setup()`` without the loader:
+ those are not given a second, different reading.
+ """
+ from soup_cli.data.formats import _convert_kto
+
+ if "label" not in row:
+ raise PreferenceRowError("the row has no 'label' (true for desirable, false for not)")
+ label = row["label"]
+ if not isinstance(label, bool):
+ raise PreferenceRowError(
+ f"label must be true or false, got {label!r} ({type(label).__name__}); "
+ "rows that bypass the loader are not converted"
+ )
+ try:
+ return _convert_kto(row)
+ except (KeyError, TypeError, ValueError) as exc:
+ raise PreferenceRowError(str(exc)) from exc
+
+
+def render_kto_text(row: dict, tokenizer: Any) -> tuple[str, str]:
+ """``(prompt, completion)`` text, as ``trl.data_utils.apply_chat_template``.
+
+ Text rows pass through. Conversational rows render the prompt with a
+ generation prompt (or ``continue_final_message`` after an assistant turn),
+ then ``prompt + completion``; the prompt is narrowed to its common prefix
+ with that rendering and the completion is what follows it.
+ """
+ prompt, completion = row["prompt"], row["completion"]
+ if not isinstance(prompt, list):
+ return prompt, completion
+ last_role = prompt[-1]["role"]
+ if last_role in ("user", "tool"):
+ kwargs = {"add_generation_prompt": True, "continue_final_message": False}
+ elif last_role == "assistant":
+ kwargs = {"add_generation_prompt": False, "continue_final_message": True}
+ else:
+ raise PreferenceRowError(f"the prompt ends with an unsupported role {last_role!r}")
+ prompt_text = tokenizer.apply_chat_template(prompt, tokenize=False, **kwargs)
+ full = tokenizer.apply_chat_template(prompt + completion, tokenize=False)
+ prompt_text = _common_prefix(prompt_text, full)
+ return prompt_text, full[len(prompt_text):]
+
+
+def process_kto_tokens(prompt_ids: list, answer_ids: list, tokenizer: Any,
+ max_length: int) -> tuple[list, list, int]:
+ """``(prompt_ids, completion_ids, answer tokens kept)`` after TRL's ``_process_tokens``.
+
+ The room is ``max_length`` less one for a BOS and one for an EOS that are
+ still to be added; a row past it loses the end of its answer. BOS is then
+ prepended to the prompt and EOS appended to the (cut) answer when missing.
+ A prompt longer than the room keeps no answer token here, where TRL's
+ negative slice would keep some and train past ``max_length``;
+ :func:`build_kto_examples` refuses such a matched row.
+ """
+ bos = getattr(tokenizer, "bos_token_id", None)
+ eos = getattr(tokenizer, "eos_token_id", None)
+ if eos is None:
+ raise PreferenceRowError("the tokenizer has no eos_token_id to end the completion")
+ prompt_ids, answer_ids = list(prompt_ids), list(answer_ids)
+ limit = int(max_length)
+ if prompt_ids and bos != prompt_ids[0]:
+ limit -= 1
+ if answer_ids and eos != answer_ids[-1]:
+ limit -= 1
+ if len(prompt_ids) + len(answer_ids) > limit:
+ answer_ids = answer_ids[: max(0, limit - len(prompt_ids))]
+ kept = len(answer_ids)
+ if bos is not None and (not prompt_ids or bos != prompt_ids[0]):
+ prompt_ids = [bos] + prompt_ids
+ if not answer_ids or eos != answer_ids[-1]:
+ answer_ids = answer_ids + [eos]
+ return prompt_ids, answer_ids, kept
+
+
+def kl_partners(n_rows: int, group_size: int) -> list:
+ """Which row's completion each row's KL row borrows.
+
+ TRL's ``_get_kl_dataset`` rotates the completions of each mapped batch by
+ one (``[last] + all[:-1]``) and ``Dataset.map(batched=True, batch_size=n)``
+ hands it consecutive groups of ``n`` rows, the last one shorter when ``n``
+ does not divide the dataset; a last group of one keeps its own completion.
+ """
+ size = max(1, int(group_size))
+ partners = []
+ for start in range(0, n_rows, size):
+ stop = min(n_rows, start + size)
+ partners.extend([stop - 1] + list(range(start, stop - 1)))
+ return partners
+
+
+def build_kto_examples(rows: list, tokenizer: Any, *, max_length: int,
+ group_size: int) -> list:
+ """Tokenize ``{prompt, completion, label}`` rows and their KL rows like TRL 0.29.1.
+
+ Raises :class:`PreferenceRowError` (prefixed ``row <index>:``) for a row
+ with no usable label or no completion token, so a run never starts on
+ shifted KL groups.
+ """
+ raw = []
+ for index, row in enumerate(rows):
+ try:
+ norm = _normalise_row(row)
+ prompt, completion = render_kto_text(norm, tokenizer)
+ prompt_ids, answer_ids = _tokenized_answer(tokenizer, prompt, completion)
+ if not answer_ids:
+ raise PreferenceRowError(
+ "the completion is empty: there is no completion token to score"
+ )
+ full_prompt, full_answer, kept = process_kto_tokens(
+ prompt_ids, answer_ids, tokenizer, max_length
+ )
+ if not kept:
+ raise PreferenceRowError(
+ f"no completion token fits in data.max_length ({max_length}) after the "
+ f"{len(full_prompt)}-token prompt"
+ )
+ except PreferenceRowError as exc:
+ raise PreferenceRowError(f"row {index}: {exc}") from exc
+ raw.append((full_prompt, full_answer, list(answer_ids), norm["label"]))
+
+ examples = []
+ for (prompt_ids, completion_ids, _, label), j in zip(raw, kl_partners(len(raw), group_size)):
+ # TRL's KL pass re-processes this (BOS-prefixed) prompt with the
+ # partner's uncut answer.
+ _, kl_ids, _ = process_kto_tokens(prompt_ids, raw[j][2], tokenizer, max_length)
+ examples.append(KtoExample(prompt_ids, completion_ids, kl_ids, label))
+ return examples
+
+
+def collate_kto(examples: list) -> tuple[list, list]:
+ """``(tokens, completion mask)``: the matched rows, then the KL rows,
+ right-padded with 0. The mask marks the completion tokens whose
+ log-probs are summed; padding is never read."""
+ seqs, masks = [], []
+ for side in ("completion_ids", "kl_completion_ids"):
+ for ex in examples:
+ completion = getattr(ex, side)
+ seqs.append(list(ex.prompt_ids) + list(completion))
+ masks.append([0] * len(ex.prompt_ids) + [1] * len(completion))
+ width = max(len(s) for s in seqs)
+ return ([s + [0] * (width - len(s)) for s in seqs],
+ [m + [0] * (width - len(m)) for m in masks])
+
+
+def sequential_batches(n_rows: int, batch_size: int, epochs: int) -> list:
+ """TRL's KTO batches: dataset order every epoch, the last partial batch kept
+ (``SequentialSampler``, ``DataLoader(drop_last=False)``)."""
+ size = max(1, int(batch_size))
+ one = [list(range(i, min(n_rows, i + size))) for i in range(0, n_rows, size)]
+ return [list(b) for _ in range(max(1, int(epochs))) for b in one]
+
+
+# --------------------------------------------------------------------------
+# Scoring and loss
+# --------------------------------------------------------------------------
+
+
+def kto_logps(model: Any, tokens: Any, mask: Any, n: int, *,
+ head_split: Any = None) -> tuple[Any, Any]:
+ """``(policy logps, policy KL logps)`` for a collated batch of ``n`` examples.
+
+ Two forwards, as TRL runs them: the KL rows first and without gradient
+ (``_compute_kl_logps`` under ``torch.no_grad``), then the matched rows.
+ """
+ import mlx.core as mx
+
+ kl = mx.stop_gradient(
+ sequence_logps(model, tokens[n:], mask[n:], head_split=head_split)
+ )
+ return sequence_logps(model, tokens[:n], mask[:n], head_split=head_split), kl
+
+
+def kto_loss(
+ policy_logps: Any,
+ policy_kl_logps: Any,
+ reference_logps: Any,
+ reference_kl_logps: Any,
+ labels: Any,
+ beta: float,
+ *,
+ desirable_weight: float = DESIRABLE_WEIGHT,
+ undesirable_weight: float = UNDESIRABLE_WEIGHT,
+) -> tuple[Any, dict]:
+ """TRL 0.29.1's ``kto_loss`` with ``loss_type="kto"``, averaged over the batch.
+
+ ``1 - sigmoid(x)`` is computed as ``sigmoid(-x)``. Metrics are sums over
+ the batch divided by its row count (``chosen_share`` is the desirable
+ fraction), so they stay finite for a single-label batch and average
+ correctly over accumulation and validation chunks; a class mean is the
+ ratio of its ``*_per_row`` metric to its share.
+ """
+ import mlx.core as mx
+
+ labels = labels.astype(mx.bool_)
+ kl = (policy_kl_logps - reference_kl_logps).astype(mx.float32).mean()
+ z = mx.maximum(mx.stop_gradient(kl), 0.0)
+ r = (policy_logps - reference_logps).astype(mx.float32)
+ chosen = desirable_weight * mx.sigmoid(-beta * (r - z))
+ rejected = undesirable_weight * mx.sigmoid(-beta * (z - r))
+ loss = mx.where(labels, chosen, rejected).mean()
+ rows = labels.size
+ is_chosen = labels.astype(mx.float32)
+ rewards = beta * mx.stop_gradient(r)
+ logps = mx.stop_gradient(policy_logps).astype(mx.float32)
+ metrics = {
+ "kl": z,
+ "chosen_share": is_chosen.mean(),
+ "chosen_rewards_per_row": (rewards * is_chosen).sum() / rows,
+ "rejected_rewards_per_row": (rewards * (1 - is_chosen)).sum() / rows,
+ "chosen_logps_per_row": (logps * is_chosen).sum() / rows,
+ "rejected_logps_per_row": (logps * (1 - is_chosen)).sum() / rows,
+ }
+ return loss, metrics
+
+
+def class_means(metrics: dict, prefix: str = "") -> dict:
+ """TRL's logged ``rewards/chosen`` etc. from the per-row metrics: each class
+ mean is its per-row sum over its share; a class with no rows is left out."""
+ share = metrics[f"{prefix}chosen_share"]
+ out = {}
+ for side, part in (("chosen", share), ("rejected", 1.0 - share)):
+ if part > 1e-12:
+ for what in ("rewards", "logps"):
+ out[f"{prefix}{what}_{side}"] = metrics[f"{prefix}{side}_{what}_per_row"] / part
+ if f"{prefix}rewards_chosen" in out and f"{prefix}rewards_rejected" in out:
+ out[f"{prefix}rewards_margins"] = (
+ out[f"{prefix}rewards_chosen"] - out[f"{prefix}rewards_rejected"]
+ )
+ return out
+
+
+# --------------------------------------------------------------------------
+# Trainer
+# --------------------------------------------------------------------------
+
+
+class MLXKTOTrainerWrapper(MLXDPOTrainerWrapper):
+ """KTO on MLX: MLX DPO's loop and reference precompute with KTO's rows and loss."""
+
+ objective = "KTO"
+ reference_unit = "row(s) and their KL rows"
+ metric_names = (
+ "kl", "chosen_share", "chosen_rewards_per_row", "rejected_rewards_per_row",
+ "chosen_logps_per_row", "rejected_logps_per_row",
+ )
+ fail_on_nonfinite = True
+
+ def _resolve_batch_size(self) -> int:
+ """Never below 2, ``auto`` included (TRL refuses 1; see #1420)."""
+ return max(KTO_MIN_BATCH, super()._resolve_batch_size())
+
+ def _plan_batches(self, n_rows: int, batch_size: int, epochs: int, seed: int) -> list:
+ """Dataset order, the last partial batch kept: TRL's sequential sampler.
+ ``seed`` does not reorder anything, as on the transformers path."""
+ return sequential_batches(n_rows, batch_size, epochs)
+
+ def _check_train_split_size(self, dataset: dict) -> None:
+ """Before the model loads: at least two rows (one KL group); fewer
+ rows than ``batch_size`` train as one partial batch, as in TRL."""
+ rows = len(dataset.get("train") or [])
+ if 0 < rows < KTO_MIN_BATCH:
+ self._check_train_rows(rows, self._resolve_batch_size(), 0)
+
+ def _check_train_rows(self, n_rows: int, batch_size: int, ties: int) -> None:
+ """Two rows or more: below ``batch_size`` they form one partial batch
+ and one KL group, as TRL trains them."""
+ if n_rows < KTO_MIN_BATCH:
+ raise ValueError(
+ f"MLX {self.objective}: the training split has {n_rows} row(s); KTO needs "
+ f"at least {KTO_MIN_BATCH}, because a row's KL completion is another "
+ "row's. Add rows."
+ )
+
+ def _on_plan_cut(self, dropped: list, kept: int, accum: int) -> None:
+ """Batches run in dataset order, so the cut always drops the same rows."""
+ rows = sorted({i for batch in dropped for i in batch})
+ first, last = kept + 1, kept + len(dropped)
+ span = f"{first}" if first == last else f"{first}-{last}"
+ console.print(
+ f"[yellow]MLX {self.objective}: gradient_accumulation_steps={accum} keeps "
+ f"{kept} of {kept + len(dropped)} batches; batch(es) {span} of the run "
+ f"({len(rows)} row(s): {', '.join(map(str, rows))}) are not trained. "
+ "Rows run in dataset order, so these are the same rows every run; "
+ "TRL instead updates on the partial window at each epoch's end.[/]"
+ )
+
+ def _examples(self, rows: list, split: str = "train") -> tuple[list, int]:
+ """Every row, tokenized with its KL row; nothing is skipped, so the KL
+ groups are TRL's. A bad row stops the run before the model trains."""
+ if not rows:
+ if split == "train":
+ raise ValueError(f"MLX {self.objective}: the train split has no rows")
+ return [], 0
+ max_length = int(getattr(self.config.data, "max_length", 2048) or 2048)
+ try:
+ examples = build_kto_examples(
+ rows, self.tokenizer, max_length=max_length,
+ group_size=self._resolve_batch_size(),
+ )
+ except PreferenceRowError as exc:
+ raise PreferenceRowError(f"{self.objective.lower()} {split} {exc}") from exc
+ return examples, 0
+
+ @staticmethod
+ def _to_arrays(examples: list) -> tuple[Any, Any]:
+ # DPO's reference precompute reads this: the matched rows' reference
+ # log-probs come back as its first half, the KL rows' as its second.
+ import mlx.core as mx
+
+ tokens, mask = collate_kto(examples)
+ return mx.array(tokens), mx.array(mask)
+
+ def _batch_arrays(self, examples: list) -> tuple:
+ import mlx.core as mx
+
+ return (*self._to_arrays(examples), mx.array([bool(e.label) for e in examples]))
+
+ def _loss(self, model: Any, arrays: tuple, split: str, rows: Any) -> tuple:
+ tokens, mask, labels = arrays
+ ref, ref_kl = (r[rows] for r in self._ref[split])
+ policy, policy_kl = kto_logps(model, tokens, mask, ref.shape[0],
+ head_split=self._head_split)
+ return kto_loss(policy, policy_kl, ref, ref_kl, labels,
+ float(self.config.training.kto_beta))
+
+ def _format_metrics(self, metrics: dict) -> str:
+ means = class_means(metrics)
+ parts = [f"kl {metrics['kl']:.4f}"]
+ for side in ("chosen", "rejected"):
+ value = means.get(f"rewards_{side}")
+ parts.append(f"rewards/{side} " + ("n/a" if value is None else f"{value:.4f}"))
+ return ", ".join(parts)
+
+ def _objective_metadata(self) -> dict:
+ return {
+ "task": "kto",
+ "kto_beta": float(self.config.training.kto_beta),
+ "kto_loss_type": KTO_LOSS_TYPE,
+ "desirable_weight": DESIRABLE_WEIGHT,
+ "undesirable_weight": UNDESIRABLE_WEIGHT,
+ "reference": "precomputed from the base model",
+ "kl_rows": "completions rotated by one within each kl_group_size rows",
+ "kl_group_size": self._resolve_batch_size(),
+ "batch_order": "dataset order (TRL's sequential sampler)",
+ }
+
+ def train(self, display=None, tracker=None, run_id=None, resume_from_checkpoint=None) -> dict:
+ result = super().train(display=display, tracker=tracker, run_id=run_id,
+ resume_from_checkpoint=resume_from_checkpoint)
+ if "val_chosen_share" in result:
+ result.update(class_means(result, prefix="val_"))
+ return result
+
+ def _load_checkpoint_weights(self, checkpoint_path: str) -> None:
+ """Warm start only from a KTO adapter of the same base.
+
+ The checkpoint's directory must hold the ``adapter_config.json`` an MLX
+ KTO run writes (before its first step, so step checkpoints have it
+ too). An adapter of another task, another base, or no metadata at all
+ is refused before any tensor loads; MLX SFT's loader then checks the
+ tensor names and shapes. The reference was already computed from the
+ base model, before this runs.
+ """
+ import json
+ from pathlib import Path
+
+ name = self.objective
+ meta_path = Path(checkpoint_path).resolve().parent / "adapter_config.json"
+ if not meta_path.is_file():
+ raise ValueError(
+ f"MLX {name}: no adapter_config.json beside {checkpoint_path}; cannot "
+ "confirm it is a task='kto' adapter of this base model"
+ )
+ try:
+ meta = json.loads(meta_path.read_text(encoding="utf-8"))
+ except (OSError, ValueError) as exc:
+ raise ValueError(f"MLX {name}: unreadable {meta_path}: {exc}") from exc
+ if meta.get("task") != "kto":
+ raise ValueError(
+ f"MLX {name}: {checkpoint_path} was trained with task="
+ f"{meta.get('task')!r}, not 'kto'; start a fresh run or resume with "
+ "the matching task"
+ )
+ if str(meta.get("model")) != str(self.config.base):
+ raise ValueError(
+ f"MLX {name}: {checkpoint_path} belongs to base {meta.get('model')!r}, "
+ f"not {self.config.base!r}"
+ )
+ super()._load_checkpoint_weights(checkpoint_path)
diff --git a/src/soup_cli/trainer/mlx_routing.py b/src/soup_cli/trainer/mlx_routing.py
index 59696961..44fd6c5f 100644
--- a/src/soup_cli/trainer/mlx_routing.py
+++ b/src/soup_cli/trainer/mlx_routing.py
@@ -28,6 +28,10 @@ class _MLXRegistry(dict):
from soup_cli.trainer.mlx_simpo import MLXSimPOTrainerWrapper
return MLXSimPOTrainerWrapper
+ if task == "kto":
+ from soup_cli.trainer.mlx_kto import MLXKTOTrainerWrapper
+
+ return MLXKTOTrainerWrapper
if task == "grpo":
from soup_cli.trainer.mlx_grpo import MLXGRPOTrainerWrapper
@@ -35,13 +39,13 @@ class _MLXRegistry(dict):
raise KeyError(task)
def __contains__(self, key: object) -> bool:
- return key in ("sft", "dpo", "orpo", "simpo", "grpo")
+ return key in ("sft", "dpo", "orpo", "simpo", "kto", "grpo")
def __getitem__(self, key: str):
return self._resolve(key)
def keys(self): # noqa: D401
- return ("sft", "dpo", "orpo", "simpo", "grpo")
+ return ("sft", "dpo", "orpo", "simpo", "kto", "grpo")
MLX_TRAINER_REGISTRY: _MLXRegistry = _MLXRegistry()
diff --git a/tests/test_issue1213_dispatch_parity.py b/tests/test_issue1213_dispatch_parity.py
index 77f16f39..d706ef12 100644
--- a/tests/test_issue1213_dispatch_parity.py
+++ b/tests/test_issue1213_dispatch_parity.py
@@ -84,6 +84,7 @@ _MLX_EXPECTED = {
"dpo": "MLXDPOTrainerWrapper",
"orpo": "MLXORPOTrainerWrapper",
"simpo": "MLXSimPOTrainerWrapper",
+ "kto": "MLXKTOTrainerWrapper",
"grpo": "MLXGRPOTrainerWrapper",
}
diff --git a/tests/test_issue1324_mlx_loraplus_refusal.py b/tests/test_issue1324_mlx_loraplus_refusal.py
index de3c86c8..6250171d 100644
--- a/tests/test_issue1324_mlx_loraplus_refusal.py
+++ b/tests/test_issue1324_mlx_loraplus_refusal.py
@@ -94,7 +94,7 @@ def test_train_dry_run_exits_nonzero_on_mlx_loraplus(tmp_path: Path):
@pytest.mark.parametrize(
"task, data_format",
- [("ipo", "dpo"), ("kto", "kto"), ("pretrain", "plaintext"), ("embedding", "alpaca")],
+ [("ipo", "dpo"), ("ppo", "dpo"), ("pretrain", "plaintext"), ("embedding", "alpaca")],
)
def test_a_task_mlx_cannot_run_is_refused_for_that_first(task, data_format):
"""The MLX task gate runs first: switching backend is the whole fix there."""
diff --git a/tests/test_issue1358_vocab_expansion_refused.py b/tests/test_issue1358_vocab_expansion_refused.py
index 1ab20e12..f43dbc5d 100644
--- a/tests/test_issue1358_vocab_expansion_refused.py
+++ b/tests/test_issue1358_vocab_expansion_refused.py
@@ -135,11 +135,13 @@ class TestTheOtherPathsThatDropThem:
assert "data.add_new_tokens is not applied on backend='mlx'" in message, message
@pytest.mark.parametrize("field", _FIELDS)
- @pytest.mark.parametrize("task", ["dpo", "orpo", "simpo"])
- def test_mlx_preference_loops(self, task, field):
- """The native DPO, ORPO and SimPO loops tokenize with the base tokenizer as
- it is; the refusal rests on each being in ``MLX_TASKS`` (review of #2073)."""
- message = _refusal(_yaml(task, backend="mlx", field=field, data={"format": "dpo"}))
+ @pytest.mark.parametrize("task, fmt", [
+ ("dpo", "dpo"), ("orpo", "dpo"), ("simpo", "dpo"), ("kto", "kto"),
+ ])
+ def test_mlx_preference_loops(self, task, fmt, field):
+ """The native DPO, ORPO, SimPO and KTO loops tokenize with the base tokenizer
+ as it is; the refusal rests on each being in ``MLX_TASKS`` (review of #2073)."""
+ message = _refusal(_yaml(task, backend="mlx", field=field, data={"format": fmt}))
assert f"data.{field} is not applied on backend='mlx'" in message, message
@@ -286,7 +288,6 @@ class TestABackendTheTaskCannotRunOnIsNamedFirst:
@pytest.mark.parametrize(("task", "backend", "expected"), [
("online_dpo", "unsloth", "task='online_dpo' requires backend='transformers'"),
("ipo", "mlx", "MLX backend only ships SFT"),
- ("kto", "mlx", "MLX backend only ships SFT"),
])
def test_the_backend_refusal_comes_first(self, task, backend, expected):
message = _refusal(_yaml(task, backend=backend))
diff --git a/tests/test_mlx_chat_template_refusal.py b/tests/test_mlx_chat_template_refusal.py
index 5ac55198..94e593fb 100644
--- a/tests/test_mlx_chat_template_refusal.py
+++ b/tests/test_mlx_chat_template_refusal.py
@@ -19,7 +19,7 @@ TEMPLATES = ["chatml", "llama3", "{% for m in messages %}{{ m['content'] }}{% en
class TestMlxChatTemplateRefusal:
- @pytest.mark.parametrize("task", ["sft", "dpo", "orpo", "simpo"])
+ @pytest.mark.parametrize("task", ["sft", "dpo", "orpo", "simpo", "kto"])
@pytest.mark.parametrize("template", TEMPLATES)
def test_mlx_with_chat_template_refused_at_config_load(self, template: str, task: str) -> None:
with pytest.raises(ValueError) as exc_info:
@@ -35,7 +35,7 @@ class TestMlxChatTemplateRefusal:
assert "backend: transformers" in message
assert "Remove data.chat_template" in message
- @pytest.mark.parametrize("task", ["sft", "dpo", "orpo", "simpo"])
+ @pytest.mark.parametrize("task", ["sft", "dpo", "orpo", "simpo", "kto"])
def test_empty_chat_template_still_loads_on_mlx(self, task: str) -> None:
# The field validator normalises "" to None, which is not an override.
config = SoupConfig(
@@ -47,7 +47,7 @@ class TestMlxChatTemplateRefusal:
)
assert config.data.chat_template is None
- @pytest.mark.parametrize("task", ["sft", "dpo", "orpo", "simpo"])
+ @pytest.mark.parametrize("task", ["sft", "dpo", "orpo", "simpo", "kto"])
@pytest.mark.parametrize("template", TEMPLATES)
def test_transformers_backend_still_accepts_chat_template(
self, template: str, task: str,
@@ -62,7 +62,7 @@ class TestMlxChatTemplateRefusal:
)
assert config.data.chat_template == template
- @pytest.mark.parametrize("task", ["sft", "dpo", "orpo", "simpo"])
+ @pytest.mark.parametrize("task", ["sft", "dpo", "orpo", "simpo", "kto"])
def test_mlx_without_chat_template_still_allowed(self, task: str) -> None:
config = SoupConfig(
base="test/model",
diff --git a/tests/test_mlx_dpo.py b/tests/test_mlx_dpo.py
index 26d43d84..f7774102 100644
--- a/tests/test_mlx_dpo.py
+++ b/tests/test_mlx_dpo.py
@@ -468,9 +468,9 @@ class TestConfig:
fields = {entry.field for entry in check_config(cfg)}
assert fields == {"training.use_galore"}
- # orpo and simpo have their own MLX trainers since #2004 / #1999
- # (tests/test_mlx_orpo.py, tests/test_mlx_simpo.py).
- @pytest.mark.parametrize("task", ["ipo", "kto", "ppo"])
+ # orpo, simpo and kto have their own MLX trainers since #2004 / #1999 / #2000
+ # (tests/test_mlx_orpo.py, tests/test_mlx_simpo.py, tests/test_mlx_kto.py).
+ @pytest.mark.parametrize("task", ["ipo", "ppo"])
def test_other_preference_tasks_still_refused(self, task):
from soup_cli.config.loader import load_config_from_string
@@ -716,6 +716,60 @@ class TestPromptCapIsWired:
assert len(example.prompt_ids) + len(example.chosen_ids) <= 40
+class TestThePlanHooksKeepDpoAndOrpoBehaviour:
+ """Pins the 5ce31005 refactor (no MLX): DPO and ORPO still plan seeded,
+ shuffled, whole batches and run ``auto`` at 1; only KTO overrides the hooks."""
+
+ def _wrapper(self, cls, yaml_text):
+ from soup_cli.config.loader import load_config_from_string
+
+ return cls(load_config_from_string(yaml_text))
+
+ def _both(self):
+ from soup_cli.trainer.mlx_dpo import MLXDPOTrainerWrapper
+ from soup_cli.trainer.mlx_orpo import MLXORPOTrainerWrapper
+ from tests.test_mlx_orpo import _ORPO_YAML
+
+ return [self._wrapper(MLXDPOTrainerWrapper, _MLX_DPO_YAML),
+ self._wrapper(MLXORPOTrainerWrapper, _ORPO_YAML)]
+
+ def test_the_plan_is_seeded_shuffled_and_drops_the_partial_batch(self):
+ for wrapper in self._both():
+ plan = wrapper._plan_batches(7, 2, 2, 3)
+ assert plan == wrapper._plan_batches(7, 2, 2, 3)
+ assert plan != wrapper._plan_batches(7, 2, 2, 4)
+ assert len(plan) == 6 and all(len(b) == 2 for b in plan)
+ for epoch in (plan[:3], plan[3:]): # 6 distinct rows; the 7th never trains
+ rows = [i for b in epoch for i in b]
+ assert len(set(rows)) == 6 and set(rows) <= set(range(7))
+ sequential = [[0, 1], [2, 3], [4, 5]] * 2
+ assert any(wrapper._plan_batches(7, 2, 2, s) != sequential for s in range(3))
+
+ def test_auto_batch_size_is_one(self):
+ for wrapper in self._both():
+ wrapper.config.training.batch_size = "auto"
+ assert wrapper._resolve_batch_size() == 1
+ wrapper.config.training.batch_size = 3
+ assert wrapper._resolve_batch_size() == 3
+
+ def test_only_kto_overrides_the_hooks(self):
+ from soup_cli.trainer.mlx_dpo import MLXDPOTrainerWrapper
+ from soup_cli.trainer.mlx_kto import MLXKTOTrainerWrapper
+ from soup_cli.trainer.mlx_orpo import MLXORPOTrainerWrapper
+
+ for hook in ("_plan_batches", "_resolve_batch_size", "_check_train_rows",
+ "_on_plan_cut"):
+ base = getattr(MLXDPOTrainerWrapper, hook)
+ assert getattr(MLXORPOTrainerWrapper, hook) is base, hook
+ assert getattr(MLXKTOTrainerWrapper, hook) is not base, hook
+
+ def test_fewer_pairs_than_a_batch_is_still_refused(self):
+ wrapper = self._both()[0]
+ with pytest.raises(ValueError, match=r"3 training pair\(s\) remain, fewer than "
+ r"training.batch_size \(4\), after skipping 1"):
+ wrapper._check_train_rows(3, 4, 1)
+
+
class TestConfigRefusals:
def _load(self, extra_top="", extra_training=""):
from soup_cli.config.loader import load_config_from_string
diff --git a/tests/test_mlx_kto.py b/tests/test_mlx_kto.py
new file mode 100644
index 00000000..661786fa
--- /dev/null
+++ b/tests/test_mlx_kto.py
@@ -0,0 +1,1255 @@
+"""Native KTO on MLX (#2000).
+
+The objective is pinned to TRL 0.29.1's ``experimental/kto/kto_trainer.py``,
+which Soup's transformers backend runs for ``task: kto`` (``trainer/kto.py``
+forwards ``kto_beta`` and ``max_length`` only, so ``loss_type``,
+``desirable_weight`` and ``undesirable_weight`` stay at ``KTOConfig``'s
+defaults). Layers:
+
+- the TRL facts the objective rests on, read from TRL itself (no MLX);
+- tokenization and the KL (mismatched) rows against TRL's own ``_tokenize`` /
+ ``_process_tokens`` / ``_get_kl_dataset`` pipeline on a real
+ ``PreTrainedTokenizerFast`` (needs ``trl`` / ``datasets``, no MLX);
+- an independent float64 NumPy oracle, tied to TRL's own ``kto_loss`` with
+ torch (no MLX) and shown to be moved by every wrong objective the issue lists;
+ the MLX loss, its terms and its gradients against that oracle (MLX);
+- training, the fixed reference, saving and a fresh-process reload on a tiny
+ random Llama (needs MLX and mlx-lm). No downloads.
+"""
+
+from __future__ import annotations
+
+import ast
+import inspect
+import json
+import math
+import subprocess
+import sys
+import textwrap
+import types
+from pathlib import Path
+
+import numpy as np
+import pytest
+
+from tests.test_mlx_dpo import _strip_ansi
+from tests.test_mlx_orpo import (
+ BOS,
+ EOS,
+ CharTokenizer,
+ MergingTokenizer,
+ _decode,
+ _flat,
+ _tiny_llama_args,
+)
+
+BETA = 0.1 # KTOConfig.beta's default and Soup's kto_beta default
+
+
+def _examples(rows, tokenizer=None, max_length=256, group_size=2):
+ from soup_cli.trainer.mlx_kto import build_kto_examples
+
+ return build_kto_examples(
+ rows, tokenizer or CharTokenizer(), max_length=max_length, group_size=group_size,
+ )
+
+
+def _one(row, **kw):
+ return _examples([row, {"prompt": "z", "completion": "w", "label": False}], **kw)[0]
+
+
+# --------------------------------------------------------------------------
+# TRL facts this objective rests on (no MLX)
+# --------------------------------------------------------------------------
+
+
+def _kto_trainer_source() -> str:
+ pytest.importorskip("trl")
+ pytest.importorskip("torch")
+ from trl.experimental.kto import kto_trainer
+
+ return inspect.getsource(kto_trainer)
+
+
+class TestTrlFacts:
+ def test_the_effective_settings_are_ktoconfig_defaults(self):
+ pytest.importorskip("trl")
+ from trl.experimental.kto import KTOConfig
+
+ from soup_cli.trainer.mlx_kto import (
+ DESIRABLE_WEIGHT,
+ KTO_LOSS_TYPE,
+ UNDESIRABLE_WEIGHT,
+ )
+
+ fields = KTOConfig.__dataclass_fields__
+ assert fields["beta"].default == BETA
+ assert fields["loss_type"].default == KTO_LOSS_TYPE == "kto"
+ assert fields["desirable_weight"].default == DESIRABLE_WEIGHT == 1.0
+ assert fields["undesirable_weight"].default == UNDESIRABLE_WEIGHT == 1.0
+ assert fields["disable_dropout"].default is True
+ # 0.29.1 has no prompt cap and no truncation mode: only max_length.
+ assert not {"max_prompt_length", "max_completion_length", "truncation_mode"} & set(fields)
+
+ def test_the_transformers_kto_path_passes_beta_and_max_length_only(self):
+ """Soup builds KTOConfig with kto_beta and max_length; the prompt-length
+ helper forwards nothing on 0.29.1, and no Soup cap touches the rows."""
+ pytest.importorskip("trl")
+ from trl.experimental.kto import KTOConfig
+
+ import soup_cli.trainer.kto as kto
+ from soup_cli.trainer._trl_compat import prompt_length_kwargs
+
+ source = Path(kto.__file__).read_text(encoding="utf-8")
+ calls = [
+ node for node in ast.walk(ast.parse(source))
+ if isinstance(node, ast.Call) and getattr(node.func, "id", "") == "KTOConfig"
+ ]
+ assert len(calls) == 1
+ names = {kw.arg for kw in calls[0].keywords}
+ assert {"beta", "max_length", "per_device_train_batch_size"} <= names
+ assert not names & {"loss_type", "desirable_weight", "undesirable_weight",
+ "precompute_ref_log_probs", "disable_dropout"}
+ assert prompt_length_kwargs(KTOConfig, 64) == {}
+ assert "enforce_preference_sequence_limit" not in source
+
+ def test_the_training_sampler_is_sequential(self):
+ """``_get_train_sampler`` (kto_trainer.py:1315-1320): TRL's KTO never
+ shuffles, so a training batch is exactly one KL group."""
+ torch = pytest.importorskip("torch")
+ pytest.importorskip("trl")
+ from trl.experimental.kto.kto_trainer import KTOTrainer
+
+ sampler = KTOTrainer._get_train_sampler(types.SimpleNamespace(), list(range(5)))
+ assert isinstance(sampler, torch.utils.data.SequentialSampler)
+
+ def test_summed_logps_batch_floor_and_kl_grouping_are_in_the_source(self):
+ source = _kto_trainer_source()
+ assert source.count("average_log_prob=False") >= 4
+ assert "if args.per_device_train_batch_size <= 1:" in source
+ assert source.count("batch_size=args.per_device_train_batch_size,") == 2
+ assert "kl = (policy_KL_logps - reference_KL_logps).mean().detach()" in source
+ assert "kl = self.accelerator.gather_for_metrics(kl).mean().clamp(min=0)" in source
+ assert "loss = losses.nanmean()" in source
+
+
+# --------------------------------------------------------------------------
+# Tokenization (TRL 0.29.1 _tokenize / _process_tokens)
+# --------------------------------------------------------------------------
+
+
+_U = {"role": "user", "content": "hi"}
+
+
+class TestTokenization:
+ def test_a_generation_prompt_the_full_render_lacks_is_narrowed_away(self):
+ """TRL keeps only the prefix the prompt and ``prompt + completion``
+ renderings share (review of #2111): a template whose generation prompt
+ adds text the full rendering does not contain must not shift the split."""
+ from soup_cli.trainer.mlx_kto import render_kto_text
+
+ class _Template:
+ def apply_chat_template(self, messages, tokenize=False, add_generation_prompt=False,
+ continue_final_message=False):
+ text = "".join(f"{m['role'][0].upper()}:{m['content']}\n" for m in messages)
+ return text + ("A:<think>\n" if add_generation_prompt else "")
+
+ row = {"prompt": [{"role": "user", "content": "q"}],
+ "completion": [{"role": "assistant", "content": "ans"}]}
+ prompt, completion = render_kto_text(row, _Template())
+ assert prompt == "U:q\nA:"
+ assert completion == "ans\n"
+
+ def test_text_row_gets_bos_on_the_prompt_and_eos_on_the_completion(self):
+ ex = _one({"prompt": "Q:", "completion": " yes", "label": True})
+ assert _decode(ex.prompt_ids) == "^Q:"
+ assert _decode(ex.completion_ids) == " yes$"
+ assert ex.label is True
+
+ def test_bos_and_eos_are_not_doubled(self):
+ class WithSpecials(CharTokenizer):
+ def encode(self, text, add_special_tokens=True):
+ ids = super().encode(text, False)
+ return ([BOS] if text.startswith("Q") else []) + ids + (
+ [EOS] if text.endswith("!") else [])
+
+ ex = _one({"prompt": "Q", "completion": "a!", "label": True}, tokenizer=WithSpecials())
+ whole = ex.prompt_ids + ex.completion_ids
+ assert whole.count(BOS) == 1 and whole.count(EOS) == 1
+
+ def test_conversational_row_renders_like_trl(self):
+ ex = _one({"prompt": [_U], "completion": [{"role": "assistant", "content": "a"}],
+ "label": False})
+ assert _decode(ex.prompt_ids) == "^<user>hi|<assistant>"
+ assert _decode(ex.completion_ids) == "a|$"
+
+ def test_prompt_ending_in_assistant_continues_the_message(self):
+ ex = _one({"prompt": [_U, {"role": "assistant", "content": "Sure,"}],
+ "completion": [{"role": "assistant", "content": "a"}], "label": True})
+ assert _decode(ex.prompt_ids) == "^<user>hi|<assistant>Sure,"
+
+ def test_rows_go_through_the_loader_converter(self):
+ """A string prompt beside a message completion is the user turn."""
+ ex = _one({"prompt": "hi", "completion": [{"role": "assistant", "content": "a"}],
+ "label": True})
+ assert _decode(ex.prompt_ids) == "^<user>hi|<assistant>"
+
+ def test_a_merged_last_prompt_token_moves_the_completion_start(self):
+ ex = _one({"prompt": "zx", "completion": "y1", "label": True},
+ tokenizer=MergingTokenizer())
+ assert ex.completion_ids[0] == MergingTokenizer.MERGED
+ assert _decode(ex.prompt_ids) == "^z"
+
+ def test_an_overlong_row_cuts_the_completion_end_and_never_the_prompt(self):
+ """TRL's rule: room is max_length minus the BOS / EOS it will add."""
+ ex = _one({"prompt": "P" * 6, "completion": "abcdefgh", "label": True}, max_length=10)
+ assert _decode(ex.prompt_ids) == "^" + "P" * 6
+ assert _decode(ex.completion_ids) == "ab$"
+ assert len(ex.prompt_ids) + len(ex.completion_ids) == 10
+
+
+class TestKLRows:
+ def _rows(self, n):
+ return [{"prompt": f"p{i}:", "completion": f" c{i}" + "x" * i, "label": i % 2 == 0}
+ for i in range(n)]
+
+ def test_partners_rotate_by_one_within_each_group(self):
+ from soup_cli.trainer.mlx_kto import kl_partners
+
+ assert kl_partners(7, 3) == [2, 0, 1, 5, 3, 4, 6]
+ assert kl_partners(4, 2) == [1, 0, 3, 2]
+ assert kl_partners(5, 2) == [1, 0, 3, 2, 4] # a last group of one is its own
+
+ def test_each_kl_row_is_its_prompt_with_the_partner_completion(self):
+ examples = _examples(self._rows(5), group_size=2)
+ for i, j in enumerate([1, 0, 3, 2, 4]):
+ assert examples[i].kl_completion_ids == examples[j].completion_ids
+ assert _decode(examples[i].prompt_ids) == f"^p{i}:"
+
+ def test_a_kl_completion_is_recut_to_its_own_prompt_room(self):
+ rows = [{"prompt": "P" * 7, "completion": "ab", "label": True},
+ {"prompt": "Q", "completion": "abcdefgh", "label": False}]
+ a, b = _examples(rows, max_length=10, group_size=2)
+ assert _decode(a.kl_completion_ids) == "a$" # room 1 after "^PPPPPPP"
+ assert _decode(b.kl_completion_ids) == "ab$"
+
+ def test_the_group_is_the_resolved_batch_size(self):
+ from soup_cli.config.loader import load_config_from_string
+ from soup_cli.trainer.mlx_kto import MLXKTOTrainerWrapper
+
+ wrapper = MLXKTOTrainerWrapper(load_config_from_string(_yaml(batch_size=3)))
+ wrapper.tokenizer = CharTokenizer()
+ examples, skipped = wrapper._examples(self._rows(7))
+ assert skipped == 0
+ got = [next(j for j, e in enumerate(examples) if e.completion_ids == x.kl_completion_ids)
+ for x in examples]
+ assert got == [2, 0, 1, 5, 3, 4, 6]
+
+
+def _hf_tokenizer():
+ pytest.importorskip("trl")
+ pytest.importorskip("datasets")
+ from tokenizers import Tokenizer, models, pre_tokenizers
+ from transformers import PreTrainedTokenizerFast
+
+ words = ["<s>", "</s>", "<pad>", "<unk>", "<", ">", "user", "assistant", "|", "Q", ":",
+ "what", "is", "two", "plus", "one", "four", "five", "it", "yes", "no", "three",
+ "the", "sum", "of", "six", "seven", "answer"]
+ tok = Tokenizer(models.WordLevel(vocab={w: i for i, w in enumerate(words)},
+ unk_token="<unk>"))
+ tok.pre_tokenizer = pre_tokenizers.Sequence(
+ [pre_tokenizers.Whitespace(), pre_tokenizers.Punctuation()]
+ )
+ fast = PreTrainedTokenizerFast(
+ tokenizer_object=tok, bos_token="<s>", eos_token="</s>",
+ pad_token="<pad>", unk_token="<unk>",
+ )
+ fast.chat_template = (
+ "{% for m in messages %}<{{ m['role'] }}> {{ m['content'] }} | {% endfor %}"
+ "{% if add_generation_prompt %}<assistant> {% endif %}"
+ )
+ return fast
+
+
+def _trl_pipeline(rows, tokenizer, max_length, batch_size):
+ """TRL 0.29.1's own KTO dataset preparation, the order KTOTrainer.__init__ runs it."""
+ import datasets
+ from trl.data_utils import maybe_apply_chat_template
+ from trl.experimental.kto.kto_trainer import _get_kl_dataset, _process_tokens, _tokenize
+
+ ds = datasets.Dataset.from_list(rows)
+ ds = ds.map(maybe_apply_chat_template, fn_kwargs={"tokenizer": tokenizer})
+ ds = ds.map(_tokenize, batched=True, fn_kwargs={"tokenizer": tokenizer})
+ kwargs = {"prefix": "", "tokenizer": tokenizer, "max_length": max_length}
+ ds = ds.map(_process_tokens, fn_kwargs=kwargs)
+ kl = ds.map(_get_kl_dataset, batched=True, batch_size=batch_size)
+ kl = kl.map(_process_tokens, fn_kwargs={**kwargs, "prefix": "KL_"},
+ remove_columns=[c for c in kl.column_names if c in ds.column_names])
+ return datasets.concatenate_datasets([ds, kl], axis=1)
+
+
+_TEXT_ROWS = [
+ {"prompt": "Q: what is two plus two", "completion": " four", "label": True},
+ {"prompt": "Q: what is one plus one", "completion": " two", "label": False},
+ {"prompt": "Q: is it five", "completion": " no it is four", "label": True},
+ {"prompt": "Q: what is three", "completion": " three", "label": False},
+ {"prompt": "Q: the sum of six", "completion": " six is the sum", "label": True},
+ {"prompt": "Q: is seven one", "completion": " no", "label": False},
+ {"prompt": "Q: what is the answer", "completion": " yes the answer is seven", "label": True},
+]
+
+
+def _chat(rows):
+ return [{"prompt": [{"role": "user", "content": r["prompt"][3:]}],
+ "completion": [{"role": "assistant", "content": r["completion"].strip()}],
+ "label": r["label"]} for r in rows]
+
+
+class TestTokensAndKLRowsMatchTrl:
+ """Distinct prompts and answers, an uneven final group (7 rows, groups of 3)."""
+
+ @pytest.mark.parametrize("rows, max_length, batch_size", [
+ (_TEXT_ROWS, 64, 3),
+ (_TEXT_ROWS, 10, 3), # several answers cut, and KL rows re-cut
+ (_TEXT_ROWS, 64, 2),
+ (_chat(_TEXT_ROWS), 64, 3),
+ ])
+ def test_ids_labels_and_kl_rows_match(self, rows, max_length, batch_size):
+ tokenizer = _hf_tokenizer()
+ trl = _trl_pipeline(rows, tokenizer, max_length, batch_size)
+ ours = _examples(rows, tokenizer, max_length=max_length, group_size=batch_size)
+ assert len(ours) == len(trl)
+ for ex, ref in zip(ours, trl):
+ assert ex.prompt_ids + ex.completion_ids == list(ref["completion_input_ids"])
+ assert ex.prompt_ids + ex.kl_completion_ids == list(ref["KL_completion_input_ids"])
+ for prefix in ("", "KL_"):
+ labels = list(ref[f"{prefix}completion_labels"])
+ assert sum(1 for x in labels if x == -100) == len(ex.prompt_ids)
+ assert ex.label is ref["label"]
+
+
+# --------------------------------------------------------------------------
+# The objective against a float64 NumPy oracle
+# --------------------------------------------------------------------------
+
+
+def _sig(x):
+ return 1.0 / (1.0 + np.exp(-x))
+
+
+def _np_kto(pol, pol_kl, ref, ref_kl, labels, beta=BETA, wd=1.0, wu=1.0, *, z=None,
+ reverse=False, matched_z=False, clamp=True, equal_classes=False):
+ """KTO as TRL 0.29.1 computes it; ``z`` frozen when given (it is detached)."""
+ labels = np.asarray(labels, dtype=bool)
+ if reverse:
+ labels = ~labels
+ if z is None:
+ raw = (pol - ref).mean() if matched_z else (pol_kl - ref_kl).mean()
+ z = max(raw, 0.0) if clamp else raw
+ r = pol - ref
+ losses = np.where(labels, wd * (1.0 - _sig(beta * (r - z))),
+ wu * (1.0 - _sig(beta * (z - r))))
+ if equal_classes:
+ return 0.5 * (losses[labels].mean() + losses[~labels].mean())
+ return losses.mean()
+
+
+def _np_z(pol_kl, ref_kl):
+ return max((pol_kl - ref_kl).mean(), 0.0)
+
+
+#: Unbalanced labels (3 desirable, 1 undesirable); summed log-probs.
+_POL = np.array([-6.2, -3.1, -8.4, -4.9])
+_POL_KL = np.array([-7.0, -5.5, -6.1, -9.3])
+_LABELS = np.array([True, False, True, True])
+_DELTA = np.array([0.8, -0.5, 1.5, -1.2]) # r = policy - reference
+
+
+def _logps_case(kl_shift):
+ """``kl_shift`` > 0 gives z = kl_shift; < 0 gives a negative raw z (clamped)."""
+ return _POL, _POL_KL, _POL - _DELTA, _POL_KL - kl_shift, _LABELS
+
+
+#: (kl_shift, beta, desirable_weight, undesirable_weight)
+_SETTINGS = [(0.7, 0.1, 1.0, 1.0), (-0.4, 0.1, 1.0, 1.0), (0.7, 0.5, 1.3, 0.7),
+ (1.9, 1.0, 1.0, 2.0)]
+
+
+class TestTheOracleMatchesTrl:
+ """No MLX: the oracle against TRL's own ``KTOTrainer.kto_loss`` (bound on a
+ stub), so a CI runner without MLX still checks what the MLX tests compare to."""
+
+ def _trl(self, pol, pol_kl, ref, ref_kl, labels, beta, wd, wu):
+ torch = pytest.importorskip("torch")
+ pytest.importorskip("trl")
+ from trl.experimental.kto.kto_trainer import KTOTrainer
+
+ stub = types.SimpleNamespace(
+ calculate_KL=True, loss_type="kto", beta=beta, desirable_weight=wd,
+ undesirable_weight=wu,
+ accelerator=types.SimpleNamespace(gather_for_metrics=lambda x: x, device="cpu"),
+ )
+ lab = torch.tensor(labels)
+ p = torch.tensor(pol, dtype=torch.float64, requires_grad=True)
+ pk = torch.tensor(pol_kl, dtype=torch.float64, requires_grad=True)
+ r = torch.tensor(ref, dtype=torch.float64)
+ rk = torch.tensor(ref_kl, dtype=torch.float64)
+ losses, chosen, rejected, kl = KTOTrainer.kto_loss(
+ stub, p[lab], p[~lab], pk, r[lab], r[~lab], rk
+ )
+ loss = losses.nanmean()
+ loss.backward()
+ return loss.item(), kl.item(), p.grad.numpy(), pk.grad, chosen, rejected
+
+ @pytest.mark.parametrize("shift, beta, wd, wu", _SETTINGS)
+ def test_loss_z_rewards_and_gradient(self, shift, beta, wd, wu):
+ pol, pol_kl, ref, ref_kl, labels = _logps_case(shift)
+ loss, kl, grad, kl_grad, chosen, rejected = self._trl(
+ pol, pol_kl, ref, ref_kl, labels, beta, wd, wu
+ )
+ assert _np_kto(pol, pol_kl, ref, ref_kl, labels, beta, wd, wu) == pytest.approx(
+ loss, rel=1e-12)
+ assert _np_z(pol_kl, ref_kl) == pytest.approx(kl, abs=1e-12)
+ assert kl_grad is None # z is detached
+ np.testing.assert_allclose(chosen.numpy(), beta * _DELTA[labels], rtol=1e-12)
+ np.testing.assert_allclose(rejected.numpy(), beta * _DELTA[~labels], rtol=1e-12)
+ eps = 1e-6
+ z = _np_z(pol_kl, ref_kl)
+ for i in range(len(pol)):
+ hi, lo = pol.copy(), pol.copy()
+ hi[i] += eps
+ lo[i] -= eps
+ fd = (_np_kto(hi, pol_kl, ref, ref_kl, labels, beta, wd, wu, z=z)
+ - _np_kto(lo, pol_kl, ref, ref_kl, labels, beta, wd, wu, z=z)) / (2 * eps)
+ assert fd == pytest.approx(grad[i], rel=1e-6, abs=1e-12), i
+
+ def test_the_negative_case_really_clamps(self):
+ _, pol_kl, _, ref_kl, _ = _logps_case(-0.4)
+ assert (pol_kl - ref_kl).mean() == pytest.approx(-0.4)
+ assert self._trl(*_logps_case(-0.4), BETA, 1.0, 1.0)[1] == 0.0
+
+ @pytest.mark.parametrize("label", [True, False])
+ def test_a_single_label_batch_is_the_mean_over_its_rows(self, label):
+ pol, pol_kl, ref, ref_kl, _ = _logps_case(0.7)
+ labels = np.array([label] * 4)
+ loss = self._trl(pol, pol_kl, ref, ref_kl, labels, BETA, 1.0, 1.0)[0]
+ assert math.isfinite(loss)
+ assert _np_kto(pol, pol_kl, ref, ref_kl, labels) == pytest.approx(loss, rel=1e-12)
+
+
+class TestTheOracleSeesEveryListedMutation:
+ @pytest.mark.parametrize("mutation", [
+ "reverse", "matched_z", "no_clamp", "equal_classes",
+ ])
+ def test_each_mutation_moves_the_loss(self, mutation):
+ case = _logps_case(-0.4 if mutation == "no_clamp" else 0.7)
+ right = _np_kto(*case)
+ flag = {"no_clamp": ("clamp", False)}.get(mutation, (mutation, True))
+ wrong = _np_kto(*case, **dict([flag]))
+ assert abs(wrong - right) > 1e-4
+
+ def test_summed_and_mean_logps_differ_on_the_fixture(self):
+ logits, tokens, mask, n = _fixture()
+ assert not np.allclose(_np_logps(logits, tokens, mask),
+ _np_logps(logits, tokens, mask, mean=True))
+
+
+def _np_logps(logits, tokens, mask, *, mean=False):
+ shifted = logits[:, :-1] - logits[:, :-1].max(-1, keepdims=True)
+ logsm = shifted - np.log(np.exp(shifted).sum(-1, keepdims=True))
+ lp = np.take_along_axis(logsm, tokens[:, 1:][..., None], -1)[..., 0]
+ m = mask[:, 1:].astype(np.float64)
+ total = (lp * m).sum(-1)
+ return total / m.sum(-1) if mean else total
+
+
+def _fixture(seed=0, vocab=12):
+ """Four examples, three desirable, unequal prompt / completion / KL lengths."""
+ from soup_cli.trainer.mlx_kto import KtoExample, collate_kto
+
+ examples = [
+ KtoExample([1, 5, 6], [7, 8, 2], [9, 10, 2], True),
+ KtoExample([1, 3], [4, 2], [7, 8, 2], False),
+ KtoExample([1, 11, 4, 6], [9, 10, 2], [4, 2], True),
+ KtoExample([1, 8], [5, 5, 6, 2], [7, 2], True),
+ ]
+ tokens, mask = (np.array(x) for x in collate_kto(examples))
+ rng = np.random.default_rng(seed)
+ logits = rng.normal(0.0, 2.0, size=(tokens.shape[0], tokens.shape[1], vocab))
+ return logits, tokens, mask, len(examples)
+
+
+class _RowKeyed:
+ """Fixed logits per distinct token row, so separate forwards of the
+ matched and KL halves each get their own rows' logits."""
+
+ def __new__(cls, logits, tokens):
+ nn = pytest.importorskip("mlx.nn")
+ mx = pytest.importorskip("mlx.core")
+ table = np.asarray(tokens)
+
+ class Model(nn.Module):
+ def __init__(self):
+ super().__init__()
+ self.logits = mx.array(logits.astype(np.float32))
+
+ def __call__(self, x):
+ rows = np.array(x)
+ width = rows.shape[1]
+ idx = [next(i for i, t in enumerate(table) if np.array_equal(t[:width], r))
+ for r in rows]
+ return self.logits[mx.array(idx), :width]
+
+ return Model()
+
+
+def _refs(logits, tokens, mask, n, shift=0.7):
+ pol = _np_logps(logits, tokens, mask)
+ return pol[:n] - _DELTA, pol[n:] - shift
+
+
+class TestTheLossMatchesTheOracle:
+ def _mlx(self, logits, tokens, mask, n, ref, ref_kl, beta=BETA, wd=1.0, wu=1.0):
+ mx = pytest.importorskip("mlx.core")
+ from soup_cli.trainer.mlx_kto import kto_logps, kto_loss
+
+ model = _RowKeyed(logits, tokens)
+ pol, pol_kl = kto_logps(model, mx.array(tokens), mx.array(mask), n)
+ labels = mx.array(_LABELS)
+ loss, metrics = kto_loss(pol, pol_kl, mx.array(ref), mx.array(ref_kl), labels, beta,
+ desirable_weight=wd, undesirable_weight=wu)
+ return np.array(pol), np.array(pol_kl), float(loss.item()), metrics
+
+ @pytest.mark.parametrize("shift, beta, wd, wu", _SETTINGS)
+ def test_terms_total_and_metrics(self, shift, beta, wd, wu):
+ logits, tokens, mask, n = _fixture()
+ ref, ref_kl = _refs(logits, tokens, mask, n, shift)
+ pol, pol_kl, loss, metrics = self._mlx(logits, tokens, mask, n, ref, ref_kl,
+ beta, wd, wu)
+ want = _np_logps(logits, tokens, mask)
+ np.testing.assert_allclose(pol, want[:n], rtol=1e-5)
+ np.testing.assert_allclose(pol_kl, want[n:], rtol=1e-5)
+ assert loss == pytest.approx(
+ _np_kto(want[:n], want[n:], ref, ref_kl, _LABELS, beta, wd, wu), rel=1e-5)
+ z = _np_z(want[n:], ref_kl)
+ assert metrics["kl"].item() == pytest.approx(z, abs=1e-5)
+ r = want[:n] - ref
+ rows = len(_LABELS)
+ assert metrics["chosen_share"].item() == pytest.approx(_LABELS.mean())
+ assert metrics["chosen_rewards_per_row"].item() == pytest.approx(
+ beta * r[_LABELS].sum() / rows, rel=1e-5)
+ assert metrics["rejected_rewards_per_row"].item() == pytest.approx(
+ beta * r[~_LABELS].sum() / rows, rel=1e-5)
+ assert metrics["chosen_logps_per_row"].item() == pytest.approx(
+ want[:n][_LABELS].sum() / rows, rel=1e-5)
+ assert metrics["rejected_logps_per_row"].item() == pytest.approx(
+ want[:n][~_LABELS].sum() / rows, rel=1e-5)
+
+ def test_the_chunked_head_gives_the_same_logps(self):
+ mx = pytest.importorskip("mlx.core")
+ from soup_cli.trainer.mlx_kto import kto_logps
+
+ logits, tokens, mask, n = _fixture()
+ model = _RowKeyed(logits, tokens)
+ split = (model, lambda h: h)
+ full = kto_logps(model, mx.array(tokens), mx.array(mask), n)
+ chunked = kto_logps(model, mx.array(tokens), mx.array(mask), n, head_split=split)
+ for x, y in zip(full, chunked):
+ assert mx.allclose(x, y, atol=1e-5).item()
+
+ @pytest.mark.parametrize("shift, beta, wd, wu", [_SETTINGS[0], _SETTINGS[2]])
+ def test_gradients_match_finite_differences_with_z_detached(self, shift, beta, wd, wu):
+ mx = pytest.importorskip("mlx.core")
+ nn = pytest.importorskip("mlx.nn")
+ from soup_cli.trainer.mlx_kto import kto_logps, kto_loss
+
+ logits, tokens, mask, n = _fixture(seed=1)
+ ref, ref_kl = _refs(logits, tokens, mask, n, shift)
+ model = _RowKeyed(logits, tokens)
+ t, m = mx.array(tokens), mx.array(mask)
+
+ def f(mod):
+ pol, pol_kl = kto_logps(mod, t, m, n)
+ return kto_loss(pol, pol_kl, mx.array(ref), mx.array(ref_kl), mx.array(_LABELS),
+ beta, desirable_weight=wd, undesirable_weight=wu)[0]
+
+ _, grads = nn.value_and_grad(model, f)(model)
+ g = np.array(grads["logits"])
+ # z is a constant: no gradient reaches the KL rows at all.
+ assert np.abs(g[n:]).max() == 0.0
+ assert np.abs(g[:n]).max() > 1e-4
+ z = _np_z(_np_logps(logits, tokens, mask)[n:], ref_kl)
+
+ def oracle(lg):
+ pol = _np_logps(lg, tokens, mask)[:n]
+ return _np_kto(pol, None, ref, None, _LABELS, beta, wd, wu, z=z)
+
+ rng = np.random.default_rng(5)
+ eps = 1e-4
+ scored = np.argwhere(mask[:n, 1:] == 1)
+ for _ in range(12):
+ r, p = scored[rng.integers(0, len(scored))]
+ idx = (r, p, rng.integers(0, logits.shape[2]))
+ hi, lo = logits.copy(), logits.copy()
+ hi[idx] += eps
+ lo[idx] -= eps
+ fd = (oracle(hi) - oracle(lo)) / (2 * eps)
+ assert g[idx] == pytest.approx(fd, abs=2e-5, rel=2e-2), idx
+
+ def test_prompt_and_padding_logits_move_nothing(self):
+ logits, tokens, mask, n = _fixture()
+ ref, ref_kl = _refs(logits, tokens, mask, n)
+ base = self._mlx(logits, tokens, mask, n, ref, ref_kl)
+ bumped = logits.copy()
+ not_scored = mask[:, 1:] == 0
+ bumped[:, :-1][not_scored] += np.random.default_rng(3).normal(
+ 0.0, 3.0, size=(int(not_scored.sum()), logits.shape[-1]))
+ moved = self._mlx(bumped, tokens, mask, n, ref, ref_kl)
+ np.testing.assert_allclose(moved[0], base[0], rtol=1e-6)
+ np.testing.assert_allclose(moved[1], base[1], rtol=1e-6)
+ assert moved[2] == pytest.approx(base[2], rel=1e-6)
+
+
+class TestEdgeValues:
+ def _loss(self, labels, shift=0.7):
+ mx = pytest.importorskip("mlx.core")
+ from soup_cli.trainer.mlx_kto import kto_loss
+
+ pol, pol_kl, ref, ref_kl, _ = _logps_case(shift)
+ return kto_loss(mx.array(pol), mx.array(pol_kl), mx.array(ref), mx.array(ref_kl),
+ mx.array(np.asarray(labels)), BETA)
+
+ @pytest.mark.parametrize("label", [True, False])
+ def test_a_single_label_batch_is_finite_and_matches(self, label):
+ labels = [label] * 4
+ loss, metrics = self._loss(labels)
+ pol, pol_kl, ref, ref_kl, _ = _logps_case(0.7)
+ assert loss.item() == pytest.approx(_np_kto(pol, pol_kl, ref, ref_kl, labels), rel=1e-5)
+ assert all(math.isfinite(v.item()) for v in metrics.values())
+
+ def test_at_the_reference_the_loss_is_one_half_minus_the_kl_shift(self):
+ """r = 0 and z = 0: every term is 1 - sigmoid(0) = 0.5."""
+ mx = pytest.importorskip("mlx.core")
+ from soup_cli.trainer.mlx_kto import kto_loss
+
+ x = mx.array([-3.0, -4.0])
+ loss, metrics = kto_loss(x, x, x, x, mx.array([True, False]), BETA)
+ assert loss.item() == pytest.approx(0.5)
+ assert metrics["kl"].item() == 0.0
+
+ def test_nan_input_is_not_a_quiet_zero(self):
+ mx = pytest.importorskip("mlx.core")
+ from soup_cli.trainer.mlx_kto import kto_loss
+
+ x = mx.array([-3.0, -4.0])
+ loss, _ = kto_loss(x * mx.array([float("nan"), 1.0]), x, x, x,
+ mx.array([True, False]), BETA)
+ assert math.isnan(loss.item())
+
+
+# --------------------------------------------------------------------------
+# Data refusals (no MLX)
+# --------------------------------------------------------------------------
+
+
+class TestLabelsAreTheLoaders:
+ def test_labels_read_through_load_dataset_match_the_transformers_path(
+ self, tmp_path, monkeypatch
+ ):
+ """``load_dataset`` runs the loader's KTO converter for every backend: the
+ transformers trainer trains on these rows' labels as they are, and MLX
+ must read exactly the same ones (no second reading, no refusal)."""
+ from soup_cli.config.schema import DataConfig
+ from soup_cli.data.loader import load_dataset
+
+ raw = [True, False, "true", 1, -1, "no", 0]
+ path = tmp_path / "kto.jsonl"
+ path.write_text("".join(
+ json.dumps({"prompt": f"p{i}:", "completion": f" c{i}", "label": label}) + "\n"
+ for i, label in enumerate(raw)
+ ), encoding="utf-8")
+ monkeypatch.chdir(tmp_path)
+ loaded = load_dataset(DataConfig(train=str(path), format="kto", val_split=0.0))
+ transformers_labels = [row["label"] for row in loaded["train"]]
+ assert transformers_labels == [True, False, True, True, False, False, False]
+ examples, _ = _wrapper_no_mlx()._examples(loaded["train"])
+ assert [e.label for e in examples] == transformers_labels
+
+
+class TestDataRefusals:
+ @pytest.mark.parametrize("label", ["maybe", None, 1, "true", 0.5])
+ def test_a_row_bypassing_the_loader_with_a_non_bool_label_is_refused(self, label):
+ """Only a guard for rows handed to the trainer without ``load_dataset``,
+ which has already read every file row's label (test above)."""
+ from soup_cli.trainer.mlx_dpo import PreferenceRowError
+
+ rows = [{"prompt": "a", "completion": "b", "label": True},
+ {"prompt": "c", "completion": "d", "label": label}]
+ with pytest.raises(PreferenceRowError, match=r"kto train row 1: .*label must be"):
+ _wrapper_no_mlx()._examples(rows)
+
+ def test_a_missing_label_is_refused(self):
+ from soup_cli.trainer.mlx_dpo import PreferenceRowError
+
+ with pytest.raises(PreferenceRowError, match=r"kto val row 0: .*no 'label'"):
+ _wrapper_no_mlx()._examples([{"prompt": "a", "completion": "b"}], "val")
+
+ def test_an_empty_completion_is_refused(self):
+ from soup_cli.trainer.mlx_dpo import PreferenceRowError
+
+ rows = [{"prompt": "a", "completion": "", "label": True},
+ {"prompt": "c", "completion": "d", "label": False}]
+ with pytest.raises(PreferenceRowError, match="row 0: the completion is empty"):
+ _wrapper_no_mlx()._examples(rows)
+
+ @pytest.mark.parametrize("prompt_len, completion", [(10, "ok"), (16, "y" * 20)],
+ ids=["fills-the-room", "longer-than-the-room"])
+ def test_a_prompt_that_fills_max_length_is_refused(self, prompt_len, completion):
+ """A prompt longer than the room must not reach TRL's negative slice,
+ which keeps the head of a long answer past ``max_length`` (review of
+ #2111): the second row would train with the clamp at 0 removed."""
+ from soup_cli.trainer.mlx_dpo import PreferenceRowError
+
+ wrapper = _wrapper_no_mlx()
+ wrapper.config.data.max_length = 12
+ rows = [{"prompt": "P" * prompt_len, "completion": completion, "label": True},
+ {"prompt": "c", "completion": "d", "label": False}]
+ with pytest.raises(PreferenceRowError, match="row 0: no completion token fits"):
+ wrapper._examples(rows)
+
+ def test_an_empty_training_split_is_refused(self):
+ with pytest.raises(ValueError, match="MLX KTO: the train split has no rows"):
+ _wrapper_no_mlx()._examples([])
+
+ def test_fewer_rows_than_batch_size_form_one_partial_batch(self):
+ """TRL trains 3 rows at batch_size 4 as one batch and one KL group."""
+ from soup_cli.trainer.mlx_kto import kl_partners, sequential_batches
+
+ wrapper = _wrapper_no_mlx()
+ wrapper.config.training.batch_size = 4
+ wrapper._check_train_split_size({"train": _rows(3)})
+ wrapper._check_train_rows(3, 4, 0)
+ assert sequential_batches(3, 4, 1) == [[0, 1, 2]]
+ assert kl_partners(3, 4) == [2, 0, 1]
+
+ @pytest.mark.parametrize("check", ["split", "rows"])
+ def test_one_row_is_refused_in_kto_words(self, check):
+ wrapper = _wrapper_no_mlx()
+ with pytest.raises(ValueError) as info:
+ if check == "split":
+ wrapper._check_train_split_size({"train": _rows(1)})
+ else:
+ wrapper._check_train_rows(1, 2, 0)
+ message = str(info.value)
+ assert "MLX KTO: the training split has 1 row(s); KTO needs at least 2" in message
+ assert "pair" not in message and "tie" not in message
+ assert "Lower training.batch_size" not in message
+
+ def test_an_empty_val_split_is_fine(self):
+ assert _wrapper_no_mlx()._examples([], "val") == ([], 0)
+
+
+def _wrapper_no_mlx():
+ from soup_cli.config.loader import load_config_from_string
+ from soup_cli.trainer.mlx_kto import MLXKTOTrainerWrapper
+
+ wrapper = MLXKTOTrainerWrapper(load_config_from_string(_KTO_YAML))
+ wrapper.tokenizer = CharTokenizer()
+ return wrapper
+
+
+class TestBatching:
+ def test_batches_run_in_dataset_order_with_the_last_partial_one(self):
+ """TRL's SequentialSampler and DataLoader(drop_last=False)."""
+ from soup_cli.trainer.mlx_kto import sequential_batches
+
+ assert sequential_batches(7, 2, 2) == [[0, 1], [2, 3], [4, 5], [6]] * 2
+ assert sequential_batches(4, 2, 1) == [[0, 1], [2, 3]]
+
+ def test_the_trainer_plans_sequential_batches(self):
+ assert _wrapper_no_mlx()._plan_batches(5, 2, 1, 123) == [[0, 1], [2, 3], [4]]
+
+ @pytest.mark.parametrize("value, expected", [("auto", 2), (2, 2), (5, 5)])
+ def test_the_batch_size_never_drops_below_two(self, value, expected):
+ wrapper = _wrapper_no_mlx()
+ wrapper.config.training.batch_size = value
+ assert wrapper._resolve_batch_size() == expected
+
+
+class TestClassMeans:
+ """The reported ``val_rewards_*`` / ``val_logps_*`` from the per-row sums (no MLX)."""
+
+ def test_each_class_mean_is_its_per_row_sum_over_its_share(self):
+ from soup_cli.trainer.mlx_kto import class_means
+
+ metrics = {
+ "val_chosen_share": 0.25,
+ "val_chosen_rewards_per_row": 0.5,
+ "val_rejected_rewards_per_row": -0.6,
+ "val_chosen_logps_per_row": -1.0,
+ "val_rejected_logps_per_row": -6.0,
+ }
+ assert class_means(metrics, prefix="val_") == {
+ "val_rewards_chosen": pytest.approx(2.0),
+ "val_rewards_rejected": pytest.approx(-0.8),
+ "val_logps_chosen": pytest.approx(-4.0),
+ "val_logps_rejected": pytest.approx(-8.0),
+ "val_rewards_margins": pytest.approx(2.8),
+ }
+
+ def test_a_split_with_one_label_reports_that_label_only(self):
+ from soup_cli.trainer.mlx_kto import class_means
+
+ metrics = {
+ "chosen_share": 0.0,
+ "chosen_rewards_per_row": 0.0,
+ "rejected_rewards_per_row": -0.6,
+ "chosen_logps_per_row": 0.0,
+ "rejected_logps_per_row": -6.0,
+ }
+ assert class_means(metrics) == {
+ "rewards_rejected": pytest.approx(-0.6),
+ "logps_rejected": pytest.approx(-6.0),
+ }
+
+
+# --------------------------------------------------------------------------
+# Training on a tiny random Llama
+# --------------------------------------------------------------------------
+
+_KTO_YAML = """
+base: tiny-base
+task: kto
+backend: mlx
+data:
+ train: ./x.jsonl
+ format: kto
+ max_length: 128
+training:
+ epochs: 1
+ lr: 5e-3
+ kto_beta: 0.1
+ quantization: none
+ lora: {r: 4, alpha: 8, dropout: 0.0}
+output: ./output
+"""
+
+
+def _rows(n=8, prefix="q"):
+ return [{"prompt": f"{prefix}{i}:", "completion": " good" if i % 2 == 0 else " bad",
+ "label": i % 2 == 0} for i in range(n)]
+
+
+def _yaml(**training):
+ """The fixture with ``training`` keys replacing or adding lines."""
+ text = _KTO_YAML
+ for key, value in training.items():
+ line = f" {key}: {value}\n"
+ own = [x for x in text.splitlines(keepends=True) if x.startswith(f" {key}:")]
+ text = text.replace(own[0], line) if own else text.replace(
+ " kto_beta: 0.1\n", " kto_beta: 0.1\n" + line)
+ return text
+
+
+def _wrapper(tmp_path, monkeypatch, model_seed=0, **training):
+ mx = pytest.importorskip("mlx.core")
+ pytest.importorskip("mlx_lm")
+ from mlx_lm.models import llama
+
+ from soup_cli.config.loader import load_config_from_string
+ from soup_cli.trainer.mlx_kto import MLXKTOTrainerWrapper
+
+ tmp_path.mkdir(parents=True, exist_ok=True)
+ monkeypatch.chdir(tmp_path)
+ training = {"batch_size": 2, "gradient_accumulation_steps": 1, **training}
+ wrapper = MLXKTOTrainerWrapper(load_config_from_string(_yaml(**training)))
+ mx.random.seed(model_seed)
+ wrapper.model = llama.Model(_tiny_llama_args())
+ wrapper.tokenizer = CharTokenizer()
+ wrapper._dataset = {"train": _rows(), "val": _rows(2, prefix="v")}
+ return wrapper
+
+
+def _base_logps(wrapper, examples, model_seed=0):
+ """The untrained base model's summed log-probs: matched rows, then KL rows."""
+ mx = pytest.importorskip("mlx.core")
+ from mlx_lm.models import llama
+
+ from soup_cli.trainer.mlx_dpo import sequence_logps
+
+ mx.random.seed(model_seed)
+ base = llama.Model(_tiny_llama_args())
+ base.eval()
+ tokens, mask = wrapper._to_arrays(examples)
+ return np.array(sequence_logps(base, tokens, mask))
+
+
+class TestTraining:
+ def test_lora_trains_and_the_base_stays_frozen(self, tmp_path, monkeypatch):
+ wrapper = _wrapper(tmp_path, monkeypatch)
+ before = _flat(wrapper.model.parameters())
+ result = wrapper.train()
+ assert result["losses"] and all(math.isfinite(x) for x in result["losses"])
+ after = _flat(wrapper.model.parameters())
+ lora_b = [v for k, v in after.items() if k.endswith("lora_b")]
+ assert lora_b and any(np.abs(v).sum() > 0 for v in lora_b)
+ for name, value in before.items():
+ moved = name.replace(".weight", ".linear.weight")
+ key = name if name in after else moved
+ np.testing.assert_array_equal(after[key], value, err_msg=name)
+
+ def test_the_reference_is_the_base_model_after_several_updates(self, tmp_path, monkeypatch):
+ from soup_cli.trainer.mlx_dpo import sequence_logps
+
+ wrapper = _wrapper(tmp_path, monkeypatch, epochs=3, warmup_ratio=0.0)
+ result = wrapper.train()
+ assert len(result["losses"]) >= 6
+ examples, _ = wrapper._examples(_rows())
+ base = _base_logps(wrapper, examples)
+ ref_matched, ref_kl = (np.array(x) for x in wrapper._ref["train"])
+ n = len(examples)
+ np.testing.assert_allclose(ref_matched, base[:n], rtol=1e-4, atol=1e-4)
+ np.testing.assert_allclose(ref_kl, base[n:], rtol=1e-4, atol=1e-4)
+ wrapper.model.eval()
+ trained = np.array(sequence_logps(wrapper.model, *wrapper._to_arrays(examples)))
+ assert np.abs(trained - base).max() > 1e-2
+
+ def test_the_preference_is_learned_and_the_metadata_written(self, tmp_path, monkeypatch):
+ wrapper = _wrapper(tmp_path, monkeypatch, epochs=6, warmup_ratio=0.0)
+ result = wrapper.train()
+ assert all(math.isfinite(x) for x in result["losses"])
+ assert result["val_rewards_chosen"] > 0 > result["val_rewards_rejected"]
+ assert result["val_rewards_margins"] == pytest.approx(
+ result["val_rewards_chosen"] - result["val_rewards_rejected"])
+ assert result["val_kl"] >= 0
+ meta = json.loads((tmp_path / "output" / "adapter_config.json").read_text())
+ assert meta["task"] == "kto" and meta["fine_tune_type"] == "lora"
+ assert (meta["kto_beta"], meta["kto_loss_type"]) == (0.1, "kto")
+ assert (meta["desirable_weight"], meta["undesirable_weight"]) == (1.0, 1.0)
+ assert meta["kl_group_size"] == 2
+ assert not list((tmp_path / "output").glob("*.partial*"))
+
+ def test_training_moves_held_out_rewards_against_the_untrained_model(
+ self, tmp_path, monkeypatch
+ ):
+ """The base model is its own control: rewards are 0 before any update."""
+ mx = pytest.importorskip("mlx.core")
+
+ wrapper = _wrapper(tmp_path, monkeypatch, epochs=6, warmup_ratio=0.0)
+ val = _rows(2, prefix="v")
+ examples, _ = wrapper._examples(val, "val")
+ control = _base_logps(wrapper, examples)
+ wrapper.train()
+ wrapper.model.eval()
+ from soup_cli.trainer.mlx_dpo import sequence_logps
+
+ tokens, mask = wrapper._to_arrays(examples)
+ trained = np.array(sequence_logps(wrapper.model, tokens, mask))
+ r = trained[:2] - control[:2]
+ assert r[0] > 0.1 > -0.1 > r[1] # desirable up, undesirable down
+ assert mx.array(r).shape == (2,)
+
+ @pytest.mark.parametrize("label", [True, False])
+ def test_single_label_data_trains_finitely(self, tmp_path, monkeypatch, label):
+ wrapper = _wrapper(tmp_path, monkeypatch)
+ wrapper._dataset["train"] = [dict(r, label=label) for r in _rows()]
+ result = wrapper.train()
+ assert all(math.isfinite(x) for x in result["losses"])
+
+ def test_the_last_partial_batch_trains_and_val_chunks_weigh_by_rows(
+ self, tmp_path, monkeypatch
+ ):
+ mx = pytest.importorskip("mlx.core")
+
+ wrapper = _wrapper(tmp_path, monkeypatch)
+ wrapper._dataset["train"] = _rows(7)
+ wrapper._dataset["val"] = _rows(3, prefix="v")
+ result = wrapper.train()
+ assert result["total_steps"] == 4 # [0,1] [2,3] [4,5] [6], as TRL
+ examples, _ = wrapper._examples(wrapper._dataset["val"], "val")
+ wrapper.model.eval()
+ chunks = [(0, 2), (2, 3)]
+ total = 0.0
+ for lo, hi in chunks:
+ arrays = wrapper._batch_arrays(examples[lo:hi])
+ loss, _ = wrapper._loss(wrapper.model, arrays, "val", mx.arange(lo, hi))
+ total += loss.item() * (hi - lo)
+ assert result["val_loss"] == pytest.approx(total / 3, rel=1e-4)
+
+ def test_an_accumulated_update_is_the_mean_of_its_micro_batch_gradients(
+ self, tmp_path, monkeypatch
+ ):
+ mx = pytest.importorskip("mlx.core")
+ nn = pytest.importorskip("mlx.nn")
+ from mlx.utils import tree_flatten
+
+ settings = dict(gradient_accumulation_steps=2, optimizer="sgd", scheduler="constant",
+ warmup_ratio=0.0, weight_decay=0.0, max_grad_norm=1000.0, seed=4,
+ lr=0.5)
+ trained = _wrapper(tmp_path / "a", monkeypatch, **settings)
+ trained._dataset = {"train": _rows(4), "val": []}
+ result = trained.train()
+ assert len(result["losses"]) == 1
+ got = _flat(trained.model.trainable_parameters())
+
+ # The same start, by hand: seed, LoRA, reference, two micro-batch gradients.
+ manual = _wrapper(tmp_path / "b", monkeypatch, **settings)
+ mx.random.seed(4)
+ examples, _ = manual._examples(_rows(4))
+ manual._apply_lora(manual.model)
+ manual._prepare_objective(examples, [], 2)
+ start = _flat(manual.model.trainable_parameters())
+
+ def micro(lo, hi):
+ def f(model):
+ return manual._loss(model, manual._batch_arrays(examples[lo:hi]), "train",
+ mx.arange(lo, hi))[0]
+
+ _, grads = nn.value_and_grad(manual.model, f)(manual.model)
+ return {k: np.array(v) for k, v in tree_flatten(grads)}
+
+ g1, g2 = micro(0, 2), micro(2, 4)
+ assert any(np.abs(g1[k]).max() > 1e-5 for k in g1)
+ for name, value in start.items():
+ want = value - 0.5 * (g1[name] + g2[name]) / 2
+ np.testing.assert_allclose(got[name], want, atol=2e-5, err_msg=name)
+
+ def test_a_nonfinite_loss_stops_the_run(self, tmp_path, monkeypatch):
+ mx = pytest.importorskip("mlx.core")
+ from soup_cli.trainer import mlx_kto
+
+ real = mlx_kto.kto_logps
+
+ def poisoned(*args, **kwargs):
+ pol, pol_kl = real(*args, **kwargs)
+ return pol * mx.array(float("nan")), pol_kl
+
+ monkeypatch.setattr(mlx_kto, "kto_logps", poisoned)
+ wrapper = _wrapper(tmp_path, monkeypatch)
+ with pytest.raises(FloatingPointError, match="MLX KTO: nonfinite loss"):
+ wrapper.train()
+
+ def test_a_bad_row_is_refused_before_any_update(self, tmp_path, monkeypatch):
+ from soup_cli.trainer.mlx_dpo import PreferenceRowError
+
+ wrapper = _wrapper(tmp_path, monkeypatch)
+ wrapper._dataset["train"] = _rows(3) + [{"prompt": "x", "completion": "", "label": True}]
+ with pytest.raises(PreferenceRowError, match="kto train row 3"):
+ wrapper.train()
+ assert not any("lora" in k for k in _flat(wrapper.model.parameters()))
+ assert not (tmp_path / "output" / "adapters.safetensors").exists()
+
+
+class TestAccumulationCut:
+ def test_the_warning_names_the_untrained_batches_and_rows(self, monkeypatch):
+ from io import StringIO
+
+ from rich.console import Console
+
+ from soup_cli.trainer import mlx_kto
+
+ buffer = StringIO()
+ monkeypatch.setattr(mlx_kto, "console", Console(file=buffer, width=400))
+ _wrapper_no_mlx()._on_plan_cut([[8, 9], [0, 1]], 8, 4)
+ text = _strip_ansi(buffer.getvalue())
+ assert "gradient_accumulation_steps=4 keeps 8 of 10 batches" in text
+ assert "batch(es) 9-10 of the run (4 row(s): 0, 1, 8, 9) are not trained" in text
+
+ def test_a_run_whose_plan_is_cut_warns_and_trains_the_kept_batches(
+ self, tmp_path, monkeypatch
+ ):
+ from io import StringIO
+
+ from rich.console import Console
+
+ from soup_cli.trainer import mlx_kto
+
+ buffer = StringIO()
+ monkeypatch.setattr(mlx_kto, "console", Console(file=buffer, width=400))
+ wrapper = _wrapper(tmp_path, monkeypatch, gradient_accumulation_steps=2)
+ wrapper._dataset = {"train": _rows(5), "val": []} # [0,1] [2,3] [4]
+ result = wrapper.train()
+ assert result["total_steps"] == 2
+ text = _strip_ansi(buffer.getvalue())
+ assert "keeps 2 of 3 batches; batch(es) 3 of the run (1 row(s): 4)" in text
+
+ def test_no_cut_no_warning(self, tmp_path, monkeypatch):
+ from io import StringIO
+
+ from rich.console import Console
+
+ from soup_cli.trainer import mlx_kto
+
+ buffer = StringIO()
+ monkeypatch.setattr(mlx_kto, "console", Console(file=buffer, width=400))
+ wrapper = _wrapper(tmp_path, monkeypatch, gradient_accumulation_steps=2)
+ wrapper._dataset = {"train": _rows(4), "val": []}
+ wrapper.train()
+ assert "are not trained" not in _strip_ansi(buffer.getvalue())
+
+ def test_three_rows_at_batch_four_train_as_one_batch(self, tmp_path, monkeypatch):
+ wrapper = _wrapper(tmp_path, monkeypatch, batch_size=4)
+ wrapper._dataset = {"train": _rows(3), "val": []}
+ result = wrapper.train()
+ assert result["total_steps"] == 1 and math.isfinite(result["losses"][0])
+
+
+class TestResume:
+ def _first_run(self, tmp_path, monkeypatch):
+ wrapper = _wrapper(tmp_path / "a", monkeypatch, epochs=3, seed=1)
+ fresh = wrapper.train()
+ return tmp_path / "a" / "output" / "adapters.safetensors", fresh
+
+ def test_resume_from_its_own_adapter_keeps_the_base_reference(self, tmp_path, monkeypatch):
+ checkpoint, fresh = self._first_run(tmp_path, monkeypatch)
+ resumed = _wrapper(tmp_path / "b", monkeypatch, seed=1)
+ result = resumed.train(resume_from_checkpoint=str(checkpoint))
+ assert result["losses"][0] < fresh["losses"][0]
+ examples, _ = resumed._examples(_rows())
+ base = _base_logps(resumed, examples)
+ np.testing.assert_allclose(np.array(resumed._ref["train"][0]), base[: len(examples)],
+ rtol=1e-4, atol=1e-4)
+
+ @pytest.mark.parametrize("change, match", [
+ ({"task": "dpo"}, "MLX KTO: .* task='dpo', not 'kto'"),
+ ({"task": "orpo"}, "task='orpo'"),
+ ({"model": "other-base"}, "belongs to base"),
+ (None, "MLX KTO: no adapter_config.json"),
+ ])
+ def test_mismatched_or_missing_metadata_is_refused(self, tmp_path, monkeypatch, change, match):
+ checkpoint, _ = self._first_run(tmp_path, monkeypatch)
+ meta = checkpoint.parent / "adapter_config.json"
+ if change is None:
+ meta.unlink()
+ else:
+ meta.write_text(json.dumps({**json.loads(meta.read_text()), **change}))
+ resumed = _wrapper(tmp_path / "b", monkeypatch, seed=1)
+ with pytest.raises(ValueError, match=match):
+ resumed.train(resume_from_checkpoint=str(checkpoint))
+
+
+class TestFreshProcessReload:
+ def test_logits_reproduce_and_the_adapter_has_an_effect(self, tmp_path, monkeypatch):
+ mx = pytest.importorskip("mlx.core")
+
+ wrapper = _wrapper(tmp_path, monkeypatch, epochs=3)
+ base_path = tmp_path / "base.safetensors"
+ wrapper.model.save_weights(str(base_path))
+ wrapper.train()
+ examples, _ = wrapper._examples(_rows(2, prefix="v"), "val")
+ tokens, _ = wrapper._to_arrays(examples)
+ wrapper.model.eval()
+ logits = wrapper.model(tokens).astype(mx.float32)
+ mx.eval(logits)
+ np.save(tmp_path / "tokens.npy", np.array(tokens))
+
+ script = textwrap.dedent(f"""
+ import sys
+ import numpy as np
+ import mlx.core as mx
+ from mlx_lm.models import llama
+ from mlx_lm.tuner.utils import load_adapters
+ args = llama.ModelArgs(model_type="llama", hidden_size=32, num_hidden_layers=2,
+ intermediate_size=64, num_attention_heads=4, num_key_value_heads=2,
+ rms_norm_eps=1e-5, vocab_size=200, tie_word_embeddings=True)
+ model = llama.Model(args)
+ model.load_weights({str(base_path)!r})
+ if sys.argv[1] == "adapter":
+ load_adapters(model, {str(tmp_path / "output")!r})
+ model.eval()
+ t = mx.array(np.load({str(tmp_path / "tokens.npy")!r}))
+ np.save(sys.argv[2], np.array(model(t).astype(mx.float32)))
+ """)
+
+ def run(mode):
+ out = tmp_path / f"{mode}.npy"
+ subprocess.run([sys.executable, "-c", script, mode, str(out)], capture_output=True,
+ text=True, check=True, timeout=300)
+ return np.load(out)
+
+ reloaded, base = run("adapter"), run("base")
+ np.testing.assert_allclose(reloaded, np.array(logits), rtol=1e-4, atol=1e-5)
+ assert np.abs(base - reloaded).max() > 1e-3
+
+
+# --------------------------------------------------------------------------
+# Config and dispatch (no MLX)
+# --------------------------------------------------------------------------
+
+
+class TestConfig:
+ def _load(self, top="", training=""):
+ """`training` lines replace the fixture's own line for the same key."""
+ from soup_cli.config.loader import load_config_from_string
+
+ text = _KTO_YAML.replace("backend: mlx\n", "backend: mlx\n" + top)
+ for line in training.splitlines(keepends=True):
+ key = line.split(":", 1)[0]
+ own = [x for x in text.splitlines(keepends=True) if x.split(":", 1)[0] == key]
+ text = text.replace(own[0], line) if own else text.replace(
+ " kto_beta: 0.1\n", " kto_beta: 0.1\n" + line
+ )
+ return load_config_from_string(text)
+
+ def test_kto_on_mlx_loads_and_is_an_mlx_task(self):
+ from soup_cli.config.schema import MLX_TASKS
+
+ cfg = self._load(training=" kto_beta: 0.3\n")
+ assert (cfg.task, cfg.backend, cfg.training.kto_beta) == ("kto", "mlx", 0.3)
+ assert "kto" in MLX_TASKS
+
+ def test_pre_quantized_4bit_loads(self):
+ assert self._load(training=" quantization: 4bit\n").training.quantization == "4bit"
+
+ def test_kto_dispatches_to_the_native_trainer(self):
+ from soup_cli.trainer.mlx_kto import MLXKTOTrainerWrapper
+ from soup_cli.trainer.mlx_routing import MLX_TRAINER_REGISTRY, get_mlx_trainer
+
+ assert "kto" in MLX_TRAINER_REGISTRY and "kto" in MLX_TRAINER_REGISTRY.keys()
+ assert get_mlx_trainer("kto") is MLXKTOTrainerWrapper
+
+ @pytest.mark.parametrize("top, training, match", [
+ ("", " quantization: 8bit\n", "quantization='8bit' is not supported by task='kto'"),
+ ("modality: vision\n", "", "modality='vision' is not supported by task='kto'"),
+ ("", " lora: {r: 0}\n", "lora.r must be at least 1 for task='kto'"),
+ ("", " loraplus_lr_ratio: 16\n", "loraplus_lr_ratio is not implemented on backend='mlx'"),
+ ("", " lora: {r: 4, use_dora: true}\n", "MLX DoRA requires task='sft'"),
+ ("", " batch_size: 1\n", "task='kto' requires training.batch_size >= 2"),
+ ])
+ def test_unsupported_combinations_are_refused_at_load(self, top, training, match):
+ with pytest.raises(Exception, match=match):
+ self._load(top, training)
+
+ def test_batch_size_auto_loads(self):
+ assert self._load(training=" batch_size: auto\n").training.batch_size == "auto"
+
+ def test_other_tasks_name_kto_among_the_mlx_ones(self):
+ from soup_cli.config.loader import load_config_from_string
+
+ with pytest.raises(Exception, match="only ships SFT, DPO, ORPO, SimPO and KTO"):
+ load_config_from_string(_KTO_YAML.replace("task: kto", "task: ipo"))
+
+ def test_the_transformers_route_is_unchanged(self):
+ from soup_cli.config.loader import load_config_from_string
+
+ cfg = load_config_from_string(_KTO_YAML.replace("backend: mlx", "backend: transformers"))
+ assert cfg.backend == "transformers" and cfg.task == "kto"
+
+ def test_doctor_reports_ignored_settings(self):
+ from soup_cli.config.backend_support import check_config
+
+ cfg = self._load(training=" use_galore: true\n")
+ assert {e.field for e in check_config(cfg)} == {"training.use_galore"}
+
+ def test_the_ignored_settings_are_printed(self, monkeypatch):
+ from io import StringIO
+
+ from rich.console import Console
+
+ from soup_cli.trainer import mlx_dpo
+ from soup_cli.trainer.mlx_kto import MLXKTOTrainerWrapper
+
+ buffer = StringIO()
+ monkeypatch.setattr(mlx_dpo, "console", Console(file=buffer, width=400))
+ MLXKTOTrainerWrapper(self._load(training=" use_galore: true\n"))._check_unsupported()
+ assert "MLX backend ignores: GaLore" in _strip_ansi(buffer.getvalue())
diff --git a/tests/test_mlx_lora_pattern_refusal.py b/tests/test_mlx_lora_pattern_refusal.py
index 7158471f..6e221ac0 100644
--- a/tests/test_mlx_lora_pattern_refusal.py
+++ b/tests/test_mlx_lora_pattern_refusal.py
@@ -26,7 +26,7 @@ from soup_cli.config.loader import load_config_from_string
from soup_cli.config.schema import SoupConfig
from tests.conftest import strip_ansi
-_MLX_TASKS = ["sft", "dpo", "orpo", "simpo"]
+_MLX_TASKS = ["sft", "dpo", "orpo", "simpo", "kto"]
_REMOVE_OR_SWITCH = "or use backend: transformers."
_YAML = """
@@ -181,7 +181,7 @@ class TestOtherRefusalsKeepTheirOwnWords:
"""Each of these configs names a problem that "use backend: transformers" would not
fix, or that has a more basic answer, so that refusal has to come first."""
- @pytest.mark.parametrize("task", ["grpo", "kto", "ipo", "pretrain"])
+ @pytest.mark.parametrize("task", ["grpo", "ipo", "pretrain"])
def test_a_task_mlx_cannot_run_is_refused_for_that_first(self, task: str) -> None:
with pytest.raises(ValidationError) as excinfo:
_config(task, rank_pattern={"q_proj": 4})
@@ -260,7 +260,7 @@ class TestOtherEntryPoints:
flat = " ".join(text.split())
assert (
"`training.lora.rank_pattern` and `training.lora.alpha_pattern` are refused "
- "on `backend: mlx` for SFT, DPO, ORPO and SimPO"
+ "on `backend: mlx` for SFT, DPO, ORPO, SimPO and KTO"
) in flat
def test_the_doctor_table_has_no_pattern_rows(self) -> None:
diff --git a/tests/test_mlx_simpo.py b/tests/test_mlx_simpo.py
index 7f9fdd0f..06ec22fc 100644
--- a/tests/test_mlx_simpo.py
+++ b/tests/test_mlx_simpo.py
@@ -527,6 +527,7 @@ class TestTraining:
assert result["val_reward_accuracy"] == pytest.approx(1.0)
assert result["val_reward_margin"] > 0
assert math.isfinite(result["val_nll_loss"])
+ assert math.isfinite(result["val_preference_loss"])
assert result["skipped_identical_pairs"] == 0
meta = json.loads((tmp_path / "output" / "adapter_config.json").read_text())
assert meta["task"] == "simpo" and meta["fine_tune_type"] == "lora"
@@ -755,7 +756,7 @@ class TestConfig:
def test_other_tasks_name_simpo_among_the_mlx_ones(self):
from soup_cli.config.loader import load_config_from_string
- with pytest.raises(Exception, match="only ships SFT, DPO, ORPO and SimPO"):
+ with pytest.raises(Exception, match="only ships SFT, DPO, ORPO, SimPO and KTO"):
load_config_from_string(_SIMPO_YAML.replace("task: simpo", "task: ipo"))
def test_the_transformers_route_is_unchanged(self):
diff --git a/tests/test_v07204.py b/tests/test_v07204.py
index e8ba59a2..e3dfa877 100644
--- a/tests/test_v07204.py
+++ b/tests/test_v07204.py
@@ -1121,12 +1121,13 @@ class TestKtoBatchIsRefusedEarly:
assert "KL term" in message and "gradient_accumulation_steps" in message
assert "streaming" not in message
- def test_mlx_kto_reports_the_backend_refusal_first(self):
+ def test_mlx_kto_keeps_the_same_floor(self):
+ """MLX KTO (#2000) builds TRL's KL rows, so batch 1 is refused there too."""
import yaml
from soup_cli.config.loader import load_config_from_string
- with pytest.raises(ValueError, match="MLX backend only ships SFT"):
+ with pytest.raises(ValueError, match="task='kto' requires training.batch_size >= 2"):
load_config_from_string(
yaml.safe_dump(
{