FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
chore(scripts): add ftw_hotfix.py to repair FTW checkpoints from older builds (#424)
Xiaoze Fan505477ab4429
diff --git a/README.md b/README.mdindex a48a8a77..d823f8fc 100644--- a/README.md+++ b/README.md@@ -55,6 +55,7 @@ For More details:- [Quick start](https://github.com/FlashML-org/FreeToken/blob/main/docs/quickstart.md)- [Supported models](https://github.com/FlashML-org/FreeToken/blob/main/docs/models.md)- [CLI reference](https://github.com/FlashML-org/FreeToken/blob/main/docs/cli.md)+- [Repairing old FTW checkpoints](https://github.com/FlashML-org/FreeToken/blob/main/docs/ftw-hotfix.md)## Citationdiff --git a/docs/cli.md b/docs/cli.mdindex 37e44335..e1eaa8aa 100644--- a/docs/cli.md+++ b/docs/cli.md@@ -155,7 +155,8 @@ Converts an HF safetensors checkpoint to FTW, FreeToken's self-containedfast-load format; point `ft serve --model` at the output dir. `--moe-backendoffload` (default) packs experts into offload banks; `--moe-backend triton`keeps them dense for resident serving. See the FTW caveats in-[models.md](models.md#notes).+[models.md](models.md#notes); FTW files from older builds can be repaired with+[scripts/ftw_hotfix.py](ftw-hotfix.md) instead of reconverting.## ft bench bwdiff --git a/docs/ftw-hotfix.md b/docs/ftw-hotfix.mdnew file mode 100644index 00000000..77362adb--- /dev/null+++ b/docs/ftw-hotfix.md@@ -0,0 +1,57 @@+# Repairing FTW checkpoints from older builds++An FTW converted by an older FreeToken build may fail to load on the current one. The fix is to+reconvert it with `ft checkpoint`. When the source checkpoint is not on disk, run+`scripts/ftw_hotfix.py` on the FTW dir instead:++```bash+# old Qwen3.6-27B-NVFP4 FTW: fetch the missing input_scale values from the Hub, patch in place+python scripts/ftw_hotfix.py --ftw ~/models/Qwen3.6-27B-NVFP4-FTW --repo nvidia/Qwen3.6-27B-NVFP4++# old DeepSeek-V4 FTW: only renames, nothing to download+python scripts/ftw_hotfix.py --ftw ~/models/DeepSeek-V4-Flash-0731-FTW++# old Qwen3.8-Flash-Next FTW: add the 47.7 GiB PLE table from a local copy of the checkpoint, into a new dir+python scripts/ftw_hotfix.py --ftw ~/models/Qwen3.8-Flash-Next-NVFP4-FTW --source ~/models/Qwen3.8-Flash-Next-NVFP4 --out ~/models/Qwen3.8-Flash-Next-NVFP4-FTW-fixed++# just show what would change+python scripts/ftw_hotfix.py --ftw ~/models/GLM-5.2-NVFP4-FTW --dry-run+```++The script needs the installed `freetoken` package. It downloads only the tensors it needs, by+byte range, never the whole checkpoint; `--revision` pins the Hub revision to read from.++## What breaks and how it is repaired++| Error at load | Checkpoints | Repair | Download |+|---|---|---|---|+| `KeyError: '...input_scale'` | ModelOpt NVFP4 exports with FP8 attention: nvidia/Qwen3.6-27B-NVFP4, RadixArk/Qwen3.8-27B-NVFP4, nvidia/Qwen3.6-35B-A3B-NVFP4 | add the missing `input_scale` scalars | a few KiB |+| `KeyError: 'model.embed.weight'` | deepseek-ai/DeepSeek-V4-Flash-0731 | rename the index entries | none |+| `RuntimeError: Unexpected keys ... .weight_scale` | nvidia/GLM-5.2-NVFP4 | dequantize the old runtime-fp8 weights back to bf16 | none |+| `PLE shard indices are not contiguous 0..N-1: []` | Qwen3.8-Flash-Next (FTWs converted before #420) | write the PLE table as `ple-table-*.safetensors` | 47.7 GiB |++The script decides by itself: it builds the current model from the FTW's `config.json`, compares+the FTW index with the tensors the model declares, and applies only the repairs that are needed.+Before writing anything it checks that the FTW matches the model (shapes, dtypes, byte counts,+shard files) and refuses one that does not. An FTW that loads as is is left untouched.++## What it writes++- In place by default. New tensors go into one appended shard, then the index is replaced+ atomically. Shards left with replaced or dropped entries (GLM-5.2) are compacted one at a time+ into new shard files, each step ending in another atomic index swap, so the FTW is loadable at+ every moment and an interrupted run is finished by running the same command again. While no+ shard has been replaced, the previous index stays as `freetoken_weight.json.bak`.+- `--out <dir>` writes a fresh, compact FTW dir (the dir must be new or empty) and leaves the+ original untouched. Shard files that an interrupted run left behind are removed by the next+ in-place repair and left alone otherwise.+- The PLE table is judged as the engine loads it. A leftover `model.safetensors.index.json` in the+ FTW dir would make the engine look for the table through that file, so the script asks you to+ remove it before it writes the table.+- The plan prints the disk space the run needs. The last line re-checks the result: declared+ tensors still missing, structural problems and shards with dead bytes must all be 0, and the PLE+ table (Flash-Next) must be `complete`. Every shard and index write is synced to disk before the+ index is replaced, so a power loss leaves either the old or the new state.+ Long phases show a progress bar; `-v` prints every step instead.++Not covered: FTWs converted from GGUF, and checkpoints outside [models.md](models.md).diff --git a/docs/models.md b/docs/models.mdindex 10edbfae..e4115356 100644--- a/docs/models.md+++ b/docs/models.md@@ -36,6 +36,8 @@ for them; other checkpoints of the same architectures work too.- `ft checkpoint` conversion is optional — it pre-converts a checkpoint intoFreeToken's fast-load format, and `ft serve --model` auto-detects the result.+- FTW files converted by builds before the quantization refactor may fail to load;+ see [ftw-hotfix.md](ftw-hotfix.md) for the affected checkpoints and the repair tool.- DeepSeek-V4 checkpoints must keep the `inference/config.json` subdir — theauthoritative model args are read from there.- Qwen3.8-Flash-Next keeps a 47.7 GiB PLE n-gram table pinned in host RAM.diff --git a/scripts/ftw_hotfix.py b/scripts/ftw_hotfix.pynew file mode 100644index 00000000..d39ed836--- /dev/null+++ b/scripts/ftw_hotfix.py@@ -0,0 +1,961 @@+"""Patch an FTW checkpoint written by an older FreeToken so the current build loads it.++ python scripts/ftw_hotfix.py --ftw <ftw_dir> [--out <new_dir>] \+ [--repo <hf_repo_id> [--revision <rev>] | --source <local_hf_dir>] [--dry-run]++The tool builds the current model on the meta device from the FTW's own config.json, diffs the+FTW index against the tensors that model declares, and repairs the differences:++- renames (DeepSeek-V4: the dense tree moved under ``model.`` and ``.scale`` became ``.weight_scale_inv``)+- fp8 weights the current model declares as bf16 (old runtime fp8 residue) are dequantized, their ``.weight_scale`` dropped+- tensors the model declares but the FTW lacks (``input_scale``) are fetched from the HF repo by byte range+- a Qwen3.8-Flash-Next FTW without the PLE n-gram table gets it written as ``ple-table-*.safetensors``++Before anything is written the FTW is checked against the model: shapes, dtypes, byte counts and+shard files must agree, or the layout is reported as unsupported.++In place, new tensors go into an appended shard and the index is swapped atomically; shards left+with unreferenced bytes are then compacted one at a time, each step ending in another atomic index+swap. An interrupted run therefore always leaves a loadable FTW, and a rerun finishes the job.+``--out`` writes a fresh, compact FTW dir instead and never touches the original.+"""++from __future__ import annotations++import argparse+import bisect+import errno+import glob+import json+import math+import os+import re+import shutil+import struct+import sys++import torch++from freetoken.checkpoint.ftw import ALIGN, FORMAT_TAG, INDEX_NAME, _align_up, _dtype_of, _dtype_str, _SHARD_FMT+from freetoken.distributed.info import set_tp_info, try_get_tp_info+from freetoken.engine.config import EngineConfig+from freetoken.engine.engine import _decode_target+from freetoken.layers import set_rope_device+from freetoken.layers.quantization.names import NameMap+from freetoken.models import create_model+from freetoken.models.register import get_model_spec+from freetoken.utils.progress import byte_bar, count_bar+from freetoken.utils.torch_utils import torch_dtype++_ST_DTYPES = {+ "F32": torch.float32, "F16": torch.float16, "BF16": torch.bfloat16, "F8_E4M3": torch.float8_e4m3fn,+ "F8_E5M2": torch.float8_e5m2, "U8": torch.uint8, "I8": torch.int8, "I32": torch.int32, "I64": torch.int64,+}+_FLOAT_DTYPES = {torch.bfloat16, torch.float16, torch.float32}+_CHUNK = 64 << 20+_SHARD_RE = re.compile(r"^freetoken-(\d{5})\.ftw$")+_PLE_INFIX = ".ple.ple_embedding.ngram_embedding."+_PLE_SHARD_RE = re.compile(r"\.ple\.ple_embedding\.ngram_embedding\.shard_(\d+)\.weight$")+_PLE_SCALE_SUFFIX = ".ple.ple_embedding.ngram_embedding.weight_scale"+_PLE_FILE_RE = re.compile(r"^ple-table-\d{5}\.safetensors$")+_PLE_FILE_BYTES = 4 << 30+# tensors an old FTW may carry that the text model does not declare; they are kept and never count as errors+_IGNORED_PREFIXES = ("vision_tower.", "embed_vision.")+_VERBOSE = False+++def log(msg: str) -> None:+ if _VERBOSE:+ print(msg, flush=True)+++def bar(total_bytes: int, desc: str):+ """A byte progress bar in normal mode; verbose mode prints per-step lines instead."""+ return None if _VERBOSE else byte_bar(total_bytes, desc)+++def tick(b, n: int) -> None:+ if b is not None:+ b.update(n)+++def done(b) -> None:+ if b is not None:+ b.close()+++def tensor_bytes(shape, dtype: torch.dtype) -> int:+ return math.prod(shape) * torch.empty((), dtype=dtype).element_size()+++# ------------------------------------------------------------------ safetensors slicing+class TensorSource:+ """Single tensors out of an HF repo (byte-range GET) or a local safetensors dir."""++ def __init__(self, repo: str | None, local: str | None, revision: str | None = None):+ assert repo or local, "need --repo or --source to fetch tensors"+ self.repo, self.local, self.revision = repo, local, revision+ self._headers: dict[str, tuple[dict, int]] = {}+ if local:+ files = sorted(os.path.basename(p) for p in glob.glob(os.path.join(local, "*.safetensors")))+ index = os.path.join(local, "model.safetensors.index.json")+ else:+ from huggingface_hub import HfApi, hf_hub_download++ repo_files = HfApi().list_repo_files(repo, revision=revision)+ files = sorted(f for f in repo_files if f.endswith(".safetensors") and "/" not in f)+ index = (hf_hub_download(repo, "model.safetensors.index.json", revision=revision)+ if "model.safetensors.index.json" in repo_files else None)+ if index and os.path.exists(index):+ with open(index) as f:+ self.weight_map = json.load(f)["weight_map"]+ else:+ self.weight_map = {name: shard for shard in files for name in self._header(shard)[0] if name != "__metadata__"}++ def has(self, name: str) -> bool:+ return name in self.weight_map++ def _range(self, shard: str, start: int, end: int) -> bytes:+ if self.local:+ with open(os.path.join(self.local, shard), "rb") as f:+ f.seek(start)+ return f.read(end - start + 1)+ import requests+ from huggingface_hub import get_token, hf_hub_url++ headers = {"Range": f"bytes={start}-{end}"}+ token = get_token()+ if token:+ headers["Authorization"] = f"Bearer {token}"+ url = hf_hub_url(self.repo, shard, revision=self.revision)+ r = requests.get(url, headers=headers, allow_redirects=True, timeout=300)+ if r.status_code != 206:+ raise RuntimeError(f"range read of {shard} failed: HTTP {r.status_code}")+ return r.content++ def _header(self, shard: str) -> tuple[dict, int]:+ if shard not in self._headers:+ n = struct.unpack("<Q", self._range(shard, 0, 7))[0]+ self._headers[shard] = (json.loads(self._range(shard, 8, 8 + n - 1)), 8 + n)+ return self._headers[shard]++ def meta(self, name: str) -> dict:+ return self._header(self.weight_map[name])[0][name]++ def nbytes(self, name: str) -> int:+ a, b = self.meta(name)["data_offsets"]+ return b - a++ def get(self, name: str) -> torch.Tensor:+ shard = self.weight_map[name]+ header, base = self._header(shard)+ meta = header[name]+ a, b = meta["data_offsets"]+ raw = bytearray(self._range(shard, base + a, base + b - 1))+ return torch.frombuffer(raw, dtype=_ST_DTYPES[meta["dtype"]]).reshape(meta["shape"])+++# ------------------------------------------------------------------ what the current build expects+def expected_tensors(ftw_dir: str, resident_experts: bool):+ if try_get_tp_info() is None:+ set_tp_info(0, 1)+ set_rope_device(torch.device("cpu"))+ kw = dict(model_path=ftw_dir, tp_info=try_get_tp_info(), dtype=torch.bfloat16)+ mc0 = EngineConfig(**kw).model_config+ is_moe = bool(getattr(mc0, "is_moe", False) or getattr(mc0, "moe_enabled", False))+ strategy = "offload" if is_moe and not resident_experts else "fused"+ cfg = EngineConfig(**kw, moe_strategy=strategy)+ object.__setattr__(cfg.model_config, "moe_strategy", strategy)+ object.__setattr__(cfg.model_config, "decode_target", _decode_target(cfg))+ with torch.device("meta"), torch_dtype(torch.bfloat16):+ model = create_model(cfg.model_config)+ state = {k: (tuple(v.shape), v.dtype) for k, v in model.state_dict().items()}+ arch = cfg.model_config.architectures[0]+ spec = get_model_spec(arch)+ name_map = NameMap(roots=spec.checkpoint_roots, segments=spec.checkpoint_segments, packed=spec.packed_modules_mapping)+ return arch, state, name_map+++# ------------------------------------------------------------------ repairs+def dsv4_rename(name: str) -> str:+ if name == "head":+ return "model.head.weight"+ if name.endswith(".scale"):+ name = name[: -len(".scale")] + ".weight_scale_inv"+ return "model." + name+++def plan(arch: str, entries: list[dict], expected: dict, name_map: NameMap):+ """Return (renames, dequants, fetches, drops, leftovers) that turn the FTW dense set into ``expected``."""+ dense = {e["name"]: e for e in entries if e["kind"] == "weight"}+ renames: dict[str, str] = {}+ if arch.startswith("DeepseekV4") and not any(n in expected for n in dense):+ for n in dense:+ new = dsv4_rename(n)+ if new not in expected:+ raise SystemExit(f"DeepSeek-V4 rename has no target for {n!r} -> {new!r}")+ renames[n] = new+ names = {renames.get(n, n): e for n, e in dense.items()}++ dequants: list[tuple[str, dict, dict]] = []+ drops: set[str] = set()+ for n, e in names.items():+ exp = expected.get(n)+ scale = names.get(n[: -len(".weight")] + ".weight_scale") if n.endswith(".weight") else None+ if (exp and exp[1] == torch.bfloat16 and e["dtype"] == "float8_e4m3fn" and tuple(e["shape"]) == exp[0]+ and scale is not None and scale["name"] not in expected and list(scale["shape"]) == [e["shape"][0]]):+ dequants.append((n, e, scale))+ drops.add(scale["name"])++ # (name, checkpoint parts): a fused module maps to several checkpoint tensors+ fetches: list[tuple[str, list[str]]] = []+ for n in (n for n in expected if n not in names):+ module, _, leaf = n.rpartition(".")+ parts = [f"{m}.{leaf}" for m in name_map.to_checkpoint(module)] if module else [n]+ fetches.append((n, parts))++ leftovers = [n for n in names if n not in expected and n not in drops]+ return renames, dequants, fetches, drops, leftovers+++def resolve_fetches(fetches, source: TensorSource) -> tuple[dict[str, list[str]], list[str]]:+ """Map each missing tensor to the source tensors it is built from; a fused scale needs every part."""+ resolved: dict[str, list[str]] = {}+ errors: list[str] = []+ for n, parts in fetches:+ if source.has(n):+ resolved[n] = [n]+ elif all(source.has(p) for p in parts):+ if len(parts) > 1 and not n.endswith(".input_scale"):+ errors.append(f"{n} maps to {len(parts)} source tensors; fusing is not supported here")+ else:+ resolved[n] = parts+ else:+ errors.append(f"no source tensor for {n}: missing {[p for p in parts if not source.has(p)]}")+ return resolved, errors+++def dequantize_rows(weight: torch.Tensor, scale: torch.Tensor) -> torch.Tensor:+ return (weight.float() * scale.float()[:, None]).to(torch.bfloat16)+++def validate_ftw(ftw_dir: str, index: dict, expected: dict, renames: dict[str, str], dequant_names: set[str]) -> list[str]:+ """Structural checks of the index and shard files, plus shape/dtype agreement with the model."""+ problems: list[str] = []+ pos = 0+ for sh in sorted(index["shards"], key=lambda s: s["global_off"]):+ if sh["global_off"] != pos:+ problems.append(f"shard {sh['file']} starts at {sh['global_off']}, expected {pos}")+ path = os.path.join(ftw_dir, sh["file"])+ if not os.path.exists(path):+ problems.append(f"shard file {sh['file']} is missing")+ elif os.path.getsize(path) != sh["nbytes"]:+ problems.append(f"shard {sh['file']} is {os.path.getsize(path)} B, index says {sh['nbytes']}")+ pos = sh["global_off"] + sh["nbytes"]+ if pos != index["total_bytes"]:+ problems.append(f"shards end at {pos}, index total_bytes is {index['total_bytes']}")+ seen: set[str] = set()+ for e in index["tensors"]:+ if e["name"] in seen:+ problems.append(f"duplicate entry {e['name']}")+ seen.add(e["name"])+ try:+ dt = _dtype_of(e["dtype"])+ except Exception:+ problems.append(f"{e['name']}: unknown dtype {e['dtype']!r}")+ continue+ if tensor_bytes(e["shape"], dt) != e["nbytes"]:+ problems.append(f"{e['name']}: {e['shape']} {e['dtype']} is {tensor_bytes(e['shape'], dt)} B, index says {e['nbytes']}")+ if e["global_off"] < 0 or e["global_off"] % ALIGN:+ problems.append(f"{e['name']}: offset {e['global_off']} is not {ALIGN}-aligned")+ if e["global_off"] + e["nbytes"] > pos:+ problems.append(f"{e['name']}: extends past the last shard")+ if e["kind"] != "weight":+ continue+ name = renames.get(e["name"], e["name"])+ exp = expected.get(name)+ if exp is None:+ continue+ if name in dequant_names: # shape already checked by plan(); the dtype is what the repair changes+ continue+ if tuple(e["shape"]) != exp[0]:+ problems.append(f"{name}: FTW shape {e['shape']} != model shape {list(exp[0])}")+ # the loader casts between float types; anything else has to match exactly+ if dt != exp[1] and not (dt in _FLOAT_DTYPES and exp[1] in _FLOAT_DTYPES):+ problems.append(f"{name}: FTW dtype {e['dtype']} is not loadable as {exp[1]}")+ blocks = sorted((e["global_off"], e["global_off"] + _align_up(e["nbytes"]), e["name"]) for e in index["tensors"])+ for (_, a1, an), (b0, _, bn) in zip(blocks, blocks[1:]):+ if b0 < a1:+ problems.append(f"{bn} overlaps {an}")+ return problems+++# ------------------------------------------------------------------ FTW I/O+class ShardWriter:+ """Streams entries into ``freetoken-NNNNN.ftw`` shards, from offset 0 (new dir) or from the old end (append)."""++ def __init__(self, out_dir: str, shard_limit: int, *, first_shard: int = 0, global_off: int = 0):+ self.out_dir, self.shard_limit = out_dir, shard_limit+ self.global_off = global_off+ self.first_shard = first_shard+ self.shard_idx = first_shard - 1+ self._f = None+ self._start = self._cur = 0+ self.shards: list[dict] = []+ self.entries: list[dict] = []++ def _path(self, idx: int) -> str:+ return os.path.join(self.out_dir, _SHARD_FMT.format(idx))++ def _finish_shard(self) -> None:+ self.shards.append({"file": _SHARD_FMT.format(self.shard_idx), "global_off": self._start, "nbytes": self._cur})+ self._f.flush()+ os.fsync(self._f.fileno())+ self._f.close()+ self._f = None++ def _roll(self) -> None:+ if self._f is not None:+ self._finish_shard()+ self.shard_idx += 1+ self._start, self._cur = self.global_off, 0+ self._f = open(self._path(self.shard_idx), "wb")++ def _write(self, data) -> None:+ off, n = 0, len(data)+ while off < n:+ if self._f is None or self._cur == self.shard_limit:+ self._roll()+ take = min(n - off, self.shard_limit - self._cur)+ self._f.write(data[off:off + take])+ off += take+ self._cur += take+ self.global_off += take++ def add(self, entry: dict, chunks) -> None:+ nbytes = entry["nbytes"]+ # a tensor that fits a shard never straddles two: roll early like FTWWriter does+ if self._f is None or (nbytes <= self.shard_limit and self._cur + nbytes > self.shard_limit):+ self._roll()+ assert self.global_off % ALIGN == 0+ self.entries.append({**entry, "global_off": self.global_off})+ written = 0+ for c in chunks:+ self._write(c)+ written += len(c)+ assert written == nbytes, (entry["name"], written, nbytes)+ pad = _align_up(self.global_off) - self.global_off+ if pad:+ self._write(bytes(pad))++ def add_tensor(self, name: str, t: torch.Tensor) -> None:+ t = t.detach().cpu().contiguous()+ raw = t.reshape(-1).view(torch.uint8).numpy().tobytes()+ self.add({"name": name, "kind": "weight", "dtype": _dtype_str(t.dtype), "shape": list(t.shape), "nbytes": len(raw)}, [raw])++ def close(self) -> list[dict]:+ if self._f is not None:+ self._finish_shard()+ return self.shards++ def abort(self) -> None:+ if self._f is not None:+ self._f.close()+ self._f = None+ for idx in range(self.first_shard, self.shard_idx + 1):+ try:+ os.remove(self._path(idx))+ except FileNotFoundError:+ pass+++def entry_chunks(ftw_dir: str, index: dict, e: dict):+ """The bytes of an existing entry, read from its shards in pieces of at most _CHUNK."""+ pos, remaining = e["global_off"], e["nbytes"]+ for sh in sorted(index["shards"], key=lambda s: s["global_off"]):+ s0, s1 = sh["global_off"], sh["global_off"] + sh["nbytes"]+ if remaining <= 0 or pos >= s1:+ continue+ with open(os.path.join(ftw_dir, sh["file"]), "rb") as f:+ f.seek(pos - s0)+ take = min(s1 - pos, remaining)+ while take > 0:+ buf = f.read(min(take, _CHUNK))+ if not buf:+ raise ValueError(f"short read in {sh['file']} for {e['name']}")+ yield buf+ take -= len(buf)+ pos += len(buf)+ remaining -= len(buf)+ assert remaining == 0, (e["name"], remaining)+++def read_entry(ftw_dir: str, index: dict, e: dict) -> torch.Tensor:+ buf = bytearray(b"".join(entry_chunks(ftw_dir, index, e)))+ return torch.frombuffer(buf, dtype=_dtype_of(e["dtype"])).reshape(e["shape"])+++def fsync_path(path: str) -> None:+ fd = os.open(path, os.O_RDONLY)+ try:+ os.fsync(fd)+ except OSError as e:+ # a filesystem may not support fsync on a directory; any other error is a real persistence failure+ if not (os.path.isdir(path) and e.errno in (errno.EINVAL, errno.ENOTSUP, errno.EOPNOTSUPP)):+ raise+ finally:+ os.close(fd)+++def write_index(ftw_dir: str, index: dict) -> None:+ # data of every file the new index points at is synced by its writer; sync the index, then the rename+ tmp = os.path.join(ftw_dir, INDEX_NAME + ".tmp")+ with open(tmp, "w") as f:+ json.dump(index, f)+ f.flush()+ os.fsync(f.fileno())+ os.replace(tmp, os.path.join(ftw_dir, INDEX_NAME))+ fsync_path(ftw_dir)+++def shard_numbers(ftw_dir: str, index: dict) -> tuple[int, list[str]]:+ """The next free shard number, and shard files on disk that the index does not reference."""+ used = {sh["file"] for sh in index["shards"]}+ on_disk = [f for f in os.listdir(ftw_dir) if _SHARD_RE.match(f)]+ nums = [int(_SHARD_RE.match(f).group(1)) for f in set(on_disk) | used if _SHARD_RE.match(f)]+ return (max(nums) + 1 if nums else 0), sorted(f for f in on_disk if f not in used)+++# ------------------------------------------------------------------ compaction+def live_ranges(tensors: list[dict], s0: int, s1: int) -> list[tuple[int, int]]:+ """Referenced byte ranges of shard [s0, s1), merged and sorted; every entry owns its ALIGN-padded block."""+ rs = []+ for e in tensors:+ a, b = e["global_off"], e["global_off"] + _align_up(e["nbytes"])+ a, b = max(a, s0), min(b, s1)+ if a < b:+ rs.append((a, b))+ rs.sort()+ merged: list[tuple[int, int]] = []+ for a, b in rs:+ if merged and a <= merged[-1][1]:+ merged[-1] = (merged[-1][0], max(merged[-1][1], b))+ else:+ merged.append((a, b))+ return merged+++def dirty_shards(index: dict) -> list[tuple[dict, list[tuple[int, int]], int]]:+ """Shards holding bytes no entry references, with their live ranges and live byte count."""+ out = []+ for sh in sorted(index["shards"], key=lambda s: s["global_off"]):+ s0, s1 = sh["global_off"], sh["global_off"] + sh["nbytes"]+ live = live_ranges(index["tensors"], s0, s1)+ live_bytes = sum(b - a for a, b in live)+ if live_bytes < sh["nbytes"]:+ out.append((sh, live, live_bytes))+ return out+++def compact_in_place(work: str, index: dict, next_num: int) -> tuple[dict, int]:+ """Rewrite each dirty shard without its dead bytes; one atomic index swap per shard."""+ dirty = dirty_shards(index)+ b = bar(sum(lb for _, _, lb in dirty), "Compacting shards") if dirty else None+ n = 0+ while dirty:+ sh, live, live_bytes = dirty[0]+ s0, s1 = sh["global_off"], sh["global_off"] + sh["nbytes"]+ delta = sh["nbytes"] - live_bytes+ old_path = os.path.join(work, sh["file"])+ new_file = None+ if live_bytes:+ new_file = _SHARD_FMT.format(next_num)+ next_num += 1+ log(f" compact {sh['file']} -> {new_file}: keep {live_bytes} B, drop {delta} B")+ with open(old_path, "rb") as src, open(os.path.join(work, new_file), "wb") as dst:+ for a, b1 in live:+ src.seek(a - s0)+ left = b1 - a+ while left > 0:+ buf = src.read(min(left, _CHUNK))+ if not buf:+ raise ValueError(f"short read in {sh['file']}")+ dst.write(buf)+ left -= len(buf)+ tick(b, len(buf))+ dst.flush()+ os.fsync(dst.fileno())+ else:+ log(f" drop empty shard {sh['file']}")+ starts = [a for a, _ in live]+ prefix = [0]+ for a, b1 in live:+ prefix.append(prefix[-1] + (b1 - a))++ def remap(go: int) -> int:+ i = bisect.bisect_right(starts, go) - 1+ if i < 0 or go >= live[i][1]:+ # only a zero-byte entry can sit in a dead gap: park it at the next live block+ return s0 + prefix[i + 1]+ return s0 + prefix[i] + (go - starts[i])++ tensors = []+ for e in index["tensors"]:+ go = e["global_off"]+ if go >= s1:+ e = {**e, "global_off": go - delta}+ elif go >= s0:+ e = {**e, "global_off": remap(go)}+ tensors.append(e)+ shards = []+ for x in sorted(index["shards"], key=lambda s: s["global_off"]):+ if x["file"] == sh["file"]:+ if new_file:+ shards.append({"file": new_file, "global_off": s0, "nbytes": live_bytes})+ elif x["global_off"] > s0:+ shards.append({**x, "global_off": x["global_off"] - delta})+ else:+ shards.append(x)+ index = {**index, "tensors": tensors, "shards": shards, "total_bytes": index["total_bytes"] - delta}+ write_index(work, index)+ os.remove(old_path)+ n += 1+ dirty = dirty_shards(index)+ done(b)+ return index, n+++# ------------------------------------------------------------------ PLE table+class PleSpec:+ """What the loader demands of the table: split_ngram_parts fp8 shards of [rows, ngram_head_dim] and one scalar scale."""++ def __init__(self, ftw_dir: str):+ with open(os.path.join(ftw_dir, "config.json")) as f:+ cfg = json.load(f)+ t = cfg.get("text_config") or cfg+ self.n_shards = int(t["split_ngram_parts"])+ self.head_dim = int(t["ple_embed_dim"]) // ((int(t["ngram_size"]) - 1) * int(t["heads_per_ngram"]))+++def _ple_collect(items, bad: list[str]) -> tuple[dict[int, tuple[tuple, int]], int | None]:+ """Group (name, dtype, shape, nbytes) rows into shard index -> (shape, nbytes) and the scale's element count."""+ shards: dict[int, tuple[tuple, int]] = {}+ scale_numel = None+ for name, dtype, shape, nbytes in items:+ if name.endswith(_PLE_SCALE_SUFFIX):+ scale_numel = math.prod(shape)+ continue+ mt = _PLE_SHARD_RE.search(name)+ if mt is None:+ continue+ idx = int(mt.group(1))+ if dtype != "F8_E4M3":+ bad.append(f"shard {idx} has dtype {dtype}")+ if idx in shards:+ bad.append(f"duplicate shard {idx}")+ shards[idx] = (tuple(shape), nbytes)+ return shards, scale_numel+++def classify_ple(spec: PleSpec, shards: dict[int, tuple[tuple, int]], scale_numel: int | None, bad: list[str]) -> tuple[str, str]:+ """('complete' | 'missing' | 'partial', detail) by the loader's rules."""+ if not shards and scale_numel is None and not bad:+ return "missing", ""+ for idx, (shape, nbytes) in sorted(shards.items()):+ if len(shape) != 2 or shape[1] != spec.head_dim:+ bad.append(f"shard {idx} is {list(shape)}, expected [rows, {spec.head_dim}]")+ elif nbytes != shape[0] * shape[1]:+ bad.append(f"shard {idx} holds {nbytes} B for shape {list(shape)}")+ if len({shape for shape, _ in shards.values()}) > 1:+ bad.append("shards differ in shape")+ if scale_numel is not None and scale_numel != 1:+ bad.append(f"weight_scale has {scale_numel} elements")+ if bad:+ return "partial", "; ".join(bad[:3])+ if scale_numel is None or sorted(shards) != list(range(spec.n_shards)):+ return "partial", f"{len(shards)}/{spec.n_shards} shards" + ("" if scale_numel is not None else ", no weight_scale")+ return "complete", ""+++def ple_source_status(source: TensorSource, spec: PleSpec) -> tuple[str, str]:+ bad: list[str] = []+ names = [k for k in source.weight_map if _PLE_INFIX in k]+ shards, scale = _ple_collect(((k, source.meta(k)["dtype"], source.meta(k)["shape"], source.nbytes(k)) for k in names), bad)+ return classify_ple(spec, shards, scale, bad)+++def ple_table_files(ftw_dir: str) -> list[str]:+ """The files the loader reads the table from: the safetensors index's mapping when there is one, else every *.safetensors."""+ index = os.path.join(ftw_dir, "model.safetensors.index.json")+ if not os.path.exists(index):+ return sorted(glob.glob(os.path.join(ftw_dir, "*.safetensors")))+ with open(index) as f:+ weight_map = json.load(f)["weight_map"]+ return sorted({os.path.join(ftw_dir, sh) for n, sh in weight_map.items() if _PLE_INFIX in n})+++def ple_table_status(ftw_dir: str, spec: PleSpec) -> tuple[str, str, list[str]]:+ """The dir's PLE table as the loader would see it: (status, detail, files holding PLE tensors)."""+ bad: list[str] = []+ items: list[tuple] = []+ files: list[str] = []+ for path in ple_table_files(ftw_dir):+ name = os.path.basename(path)+ if not os.path.exists(path):+ bad.append(f"model.safetensors.index.json maps PLE tensors to missing {name}")+ continue+ try:+ with open(path, "rb") as f:+ n = struct.unpack("<Q", f.read(8))[0]+ header, base = json.loads(f.read(n)), 8 + n+ except (struct.error, ValueError):+ bad.append(f"{name} has an unreadable header")+ files.append(name)+ continue+ size = os.path.getsize(path)+ holds = False+ for k, m in header.items():+ if k == "__metadata__" or _PLE_INFIX not in k:+ continue+ holds = True+ a, b = m["data_offsets"]+ if base + b > size:+ bad.append(f"{name} is truncated")+ items.append((k, m["dtype"], m["shape"], b - a))+ if holds:+ files.append(name)+ shards, scale = _ple_collect(items, bad)+ status, detail = classify_ple(spec, shards, scale, bad)+ return status, detail, files+++def check_ftw(ftw_dir: str, index: dict, expected: dict, renames: dict[str, str], dequant_names: set[str], ple: PleSpec | None) -> dict:+ """Everything the tool verifies, on the input before planning and on the result after writing."""+ have = {renames.get(e["name"], e["name"]) for e in index["tensors"] if e["kind"] == "weight"}+ return {+ "problems": validate_ftw(ftw_dir, index, expected, renames, dequant_names),+ "missing": [n for n in expected if n not in have],+ "extra": [n for n in have if n not in expected and not n.startswith(_IGNORED_PREFIXES)],+ "dead": dirty_shards(index),+ "ple": ple_table_status(ftw_dir, ple) if ple else ("n/a", "", []),+ }+++def ple_shards(out_dir: str, source: TensorSource) -> list[str]:+ """Write the PLE n-gram table tensors, and nothing else, into ``ple-table-*.safetensors`` files in ``out_dir``."""+ from safetensors.torch import save_file++ todo = sorted(n for n in source.weight_map if _PLE_INFIX in n)+ if not todo:+ raise SystemExit("ERROR: the source checkpoint has no PLE table tensors")+ for stale in glob.glob(os.path.join(out_dir, "ple-table-*.safetensors.tmp")):+ os.remove(stale)+ names: list[str] = []+ batch: dict[str, torch.Tensor] = {}+ size = 0+ b = bar(sum(source.nbytes(n) for n in todo), "Writing PLE table")++ def flush():+ nonlocal batch, size+ if not batch:+ return+ name = f"ple-table-{len(names):05d}.safetensors"+ log(f" write {name}: {len(batch)} tensors, {size / 2**30:.2f} GiB")+ save_file(batch, os.path.join(out_dir, name + ".tmp"))+ names.append(name)+ batch, size = {}, 0++ try:+ for n in todo:+ t = source.get(n)+ batch[n] = t+ size += t.numel() * t.element_size()+ tick(b, t.numel() * t.element_size())+ if size >= _PLE_FILE_BYTES:+ flush()+ flush()+ except BaseException:+ for name in names:+ os.remove(os.path.join(out_dir, name + ".tmp"))+ raise+ done(b)+ # every file is complete and synced before any gets its final name, so a scan never sees a half table+ for name in names:+ fsync_path(os.path.join(out_dir, name + ".tmp"))+ os.replace(os.path.join(out_dir, name + ".tmp"), os.path.join(out_dir, name))+ fsync_path(out_dir)+ return names+++def side_files(src: str, skip) -> list[str]:+ """Everything in the FTW dir except the shards and the index: configs, tokenizer, PLE tables, nested dirs."""+ return [f for f in os.listdir(src) if not (f.endswith((".ftw", ".bak", ".tmp")) or f == INDEX_NAME or skip(f))]+++def copy_side_files(src: str, dst: str, skip) -> None:+ os.makedirs(dst, exist_ok=True)+ for f in side_files(src, skip):+ s = os.path.join(src, f)+ if os.path.isdir(s):+ shutil.copytree(s, os.path.join(dst, f), dirs_exist_ok=True)+ else:+ shutil.copy2(s, os.path.join(dst, f))+++def side_files_bytes(src: str, skip) -> int:+ total = 0+ for f in side_files(src, skip):+ p = os.path.join(src, f)+ total += sum(os.path.getsize(os.path.join(r, x)) for r, _, fs in os.walk(p) for x in fs) if os.path.isdir(p) else os.path.getsize(p)+ return total+++# ------------------------------------------------------------------ main+def main(argv: list[str] | None = None) -> int:+ p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)+ p.add_argument("--ftw", required=True, help="FTW checkpoint dir to repair")+ p.add_argument("--out", help="write a fresh, compact FTW dir here (new or empty) instead of patching in place")+ p.add_argument("--repo", help="HF repo id of the source checkpoint (tensors are read by byte range)")+ p.add_argument("--revision", help="HF revision (branch, tag or commit) of --repo")+ p.add_argument("--source", help="local HF safetensors dir of the source checkpoint (instead of --repo)")+ p.add_argument("--dry-run", action="store_true")+ p.add_argument("-v", "--verbose", action="store_true", help="print every step")+ ns = p.parse_args(argv)+ global _VERBOSE+ _VERBOSE = ns.verbose++ if not os.path.isfile(os.path.join(ns.ftw, INDEX_NAME)):+ print(f"ERROR: {ns.ftw} has no {INDEX_NAME}; not an FTW checkpoint dir", file=sys.stderr)+ return 2+ with open(os.path.join(ns.ftw, INDEX_NAME)) as f:+ index = json.load(f)+ if index.get("format") != FORMAT_TAG:+ print(f"ERROR: {ns.ftw} is not an FTW checkpoint (format {index.get('format')!r})", file=sys.stderr)+ return 2+ if ns.out:+ out, ftw = os.path.abspath(ns.out), os.path.abspath(ns.ftw)+ if out == ftw or os.path.commonpath([out, ftw]) == ftw:+ print("ERROR: --out must be outside --ftw (omit --out to patch in place)", file=sys.stderr)+ return 2+ if os.path.exists(out) and not os.path.isdir(out):+ print(f"ERROR: --out {ns.out} exists and is not a directory", file=sys.stderr)+ return 2+ if os.path.isdir(out) and os.listdir(out):+ print(f"ERROR: --out {ns.out} exists and is not empty", file=sys.stderr)+ return 2+ if ns.revision and not ns.repo:+ print(" note: --revision only applies to --repo; ignored", file=sys.stderr)+ resident_experts = any(e["kind"] == "weight" and ".experts." in e["name"] for e in index["tensors"])+ log(f"reading {os.path.join(ns.ftw, INDEX_NAME)}: {len(index['tensors'])} entries, {len(index['shards'])} shards, {index['total_bytes'] / 2**30:.2f} GiB")+ log("building the current model on the meta device from the FTW's config.json" + (" (resident experts)" if resident_experts else ""))+ arch, expected, name_map = expected_tensors(ns.ftw, resident_experts)+ log(f"{arch}: model declares {len(expected)} dense tensors")+ source = TensorSource(ns.repo, ns.source, ns.revision) if (ns.repo or ns.source) else None+ renames, dequants, fetches, drops, leftovers = plan(arch, index["tensors"], expected, name_map)+ dequant_names = {n for n, _, _ in dequants}+ ple = PleSpec(ns.ftw) if arch.startswith("Qwen4Exp") else None+ chk = check_ftw(ns.ftw, index, expected, renames, dequant_names, ple)+ problems, dirty = chk["problems"], chk["dead"]+ ple_status, ple_detail, ple_files = chk["ple"]+ need_ple = ple_status in ("missing", "partial")+ next_num, orphans = shard_numbers(ns.ftw, index)++ print(f"{arch}: {len(expected)} expected dense tensors, FTW has {sum(e['kind'] == 'weight' for e in index['tensors'])}")+ print(f" renames {len(renames)} dequantize {len(dequants)} fetch {len(fetches)} drop {len(drops)} leftover {len(leftovers)}"+ + (f" PLE table: {ple_status}" + (f" ({ple_detail})" if ple_detail else "") if need_ple else "")+ + (f" dead bytes in {len(dirty)} shard(s)" if dirty else ""))+ for n, parts in fetches[:8]:+ print(f" fetch {n} <- {parts}")+ if len(fetches) > 8:+ print(f" ... {len(fetches) - 8} more")+ for old, new in list(renames.items())[:5]:+ log(f" rename {old} -> {new}")+ if len(renames) > 5:+ log(f" ... {len(renames) - 5} more renames")+ for n, e, scale_e in dequants:+ log(f" dequantize {n} (drop {scale_e['name']})")+ for n in leftovers[:8]:+ print(f" leftover (not declared by the model): {n}")+ if orphans:+ removing = not ns.out and bool(renames or dequants or fetches or drops or need_ple or dirty)+ print(f" note: {len(orphans)} shard file(s) not referenced by the index, left by an interrupted run"+ + (", will be removed: " if removing else ", left as is: ") + str(orphans[:3]))++ bad = False+ for msg in problems[:10]:+ print(f"ERROR: {msg}", file=sys.stderr)+ bad = True+ if len(problems) > 10:+ print(f"ERROR: ... {len(problems) - 10} more", file=sys.stderr)+ if (fetches or need_ple) and source is None:+ print("ERROR: tensors must be fetched but neither --repo nor --source was given", file=sys.stderr)+ return 2+ fetch_srcs: dict[str, list[str]] = {}+ if fetches:+ fetch_srcs, errors = resolve_fetches(fetches, source)+ for msg in errors:+ print(f"ERROR: {msg}", file=sys.stderr)+ bad = True+ # the converter transforms most tensors on the way in (norm offsets, fusion, packing); only the+ # activation scale scalars are stored as the checkpoint has them, so only they can be fetched raw+ unfetchable = [n for n, _ in fetches if not n.endswith(".input_scale")]+ if unfetchable:+ print(f"ERROR: {len(unfetchable)} missing tensor(s) cannot be fetched raw (first: {unfetchable[0]}); reconvert the checkpoint", file=sys.stderr)+ bad = True+ real_leftovers = [n for n in leftovers if not n.startswith(_IGNORED_PREFIXES)]+ if real_leftovers:+ print(f"ERROR: the FTW holds {len(real_leftovers)} tensors the current model does not declare (first: {real_leftovers[0]}); this layout is not supported", file=sys.stderr)+ bad = True+ foreign_ple = [f for f in ple_files if not _PLE_FILE_RE.match(f)]+ if ple_status == "partial" and foreign_ple:+ print(f"ERROR: the PLE table in {ns.ftw} is incomplete ({ple_detail}) and lives in files this tool did not write: {foreign_ple[:3]}; remove them first", file=sys.stderr)+ bad = True+ if need_ple and os.path.exists(os.path.join(ns.ftw, "model.safetensors.index.json")):+ print(f"ERROR: {ns.ftw}/model.safetensors.index.json makes the loader read the PLE table through its mapping, which cannot list the files this tool writes; remove it (an FTW dir does not need it) and rerun", file=sys.stderr)+ bad = True+ if need_ple and source is not None:+ src_status, src_detail = ple_source_status(source, ple)+ if src_status != "complete":+ print(f"ERROR: the PLE table in the source is {src_status}" + (f" ({src_detail})" if src_detail else ""), file=sys.stderr)+ bad = True+ if bad:+ return 2++ replaced = dequant_names | set(fetch_srcs)+ keep: list[tuple[dict, str]] = []+ for e in index["tensors"]:+ name = renames.get(e["name"], e["name"]) if e["kind"] == "weight" else e["name"]+ if e["kind"] == "weight" and (name in drops or name in replaced):+ continue+ keep.append((e, name))+ kept_names = {e["name"] for e, _ in keep}+ skip_ple = (lambda f: bool(_PLE_FILE_RE.match(f))) if need_ple else (lambda f: False)++ # disk: the new entries and the PLE table, plus either one compacted shard next to its original+ # (in place) or the whole compact copy (--out)+ new_bytes = sum(_align_up(tensor_bytes(*expected[n])) for n in replaced)+ table_bytes = sum(source.nbytes(n) for n in source.weight_map if _PLE_INFIX in n) if need_ple else 0+ work = ns.out or ns.ftw+ if ns.out:+ need = sum(_align_up(e["nbytes"]) for e, _ in keep) + new_bytes + table_bytes + side_files_bytes(ns.ftw, skip_ple)+ else:+ after = {**index, "tensors": [e for e in index["tensors"] if e["name"] in kept_names]}+ need = new_bytes + table_bytes + max([lb for _, _, lb in dirty_shards(after)] + [0])+ probe = os.path.abspath(work)+ while not os.path.isdir(probe):+ probe = os.path.dirname(probe)+ free = shutil.disk_usage(probe).free+ print(f" disk: about {need / 2**30:.1f} GiB needed under {work} ({free / 2**30:.1f} GiB free)")+ if ns.dry_run:+ return 0+ if not (renames or dequants or fetch_srcs or drops or need_ple or dirty or ns.out):+ print("nothing to do; the FTW loads as is")+ return 0+ if free < need + (1 << 30):+ print(f"ERROR: not enough free space under {work}", file=sys.stderr)+ return 2++ def write_new(w: ShardWriter) -> None:+ b = bar(sum(e["nbytes"] for _, e, _ in dequants), "Dequantizing") if dequants else None+ for n, e, scale_e in dequants:+ log(f" dequantize {n}: fp8 x {scale_e['name']} -> bf16 {e['shape']}")+ w.add_tensor(n, dequantize_rows(read_entry(ns.ftw, index, e), read_entry(ns.ftw, index, scale_e)))+ tick(b, e["nbytes"])+ done(b)+ items = list(fetch_srcs.items())+ for n, srcs in (items if _VERBOSE or not items else count_bar(items, "Fetching tensors")):+ log(f" fetch {n} <- {', '.join(srcs)}")+ vals = [source.get(c) for c in srcs]+ # a fused projection shares one activation scale; the reader takes the max over its parts+ w.add_tensor(n, torch.stack([v.reshape(()).float() for v in vals]).max().reshape(()))++ hotfix = {"from": os.path.abspath(ns.ftw), "renamed": len(renames), "dequantized": len(dequants),+ "fetched": len(fetch_srcs), "dropped": len(drops), "compacted": 0, "source": ns.repo or ns.source}+ compacted = 0+ if ns.out:+ log(f"--out: writing a fresh FTW -> {work}")+ try:+ copy_side_files(ns.ftw, work, skip_ple)+ w = ShardWriter(work, index["shard_limit"])+ b = bar(sum(e["nbytes"] for e, _ in keep), "Rewriting shards")+ for e, name in keep:+ log(f" copy {e['name']}" + (f" -> {name}" if name != e["name"] else "") + f" {e['nbytes']} B")+ w.add({**e, "name": name}, entry_chunks(ns.ftw, index, e))+ tick(b, e["nbytes"])+ done(b)+ write_new(w)+ shards = w.close()+ side = [f for f in index.get("side_files", []) if not skip_ple(f)]+ if need_ple:+ got = ple_shards(work, source)+ side += got+ print(f" PLE table written: {len(got)} files")+ new_index = {**index, "tensors": w.entries, "shards": shards, "total_bytes": w.global_off,+ "counts": {**index.get("counts", {}), "weight": sum(e["kind"] == "weight" for e in w.entries)},+ "side_files": sorted(side), "hotfix": hotfix}+ write_index(work, new_index)+ except BaseException:+ # the dir was new or empty, so nothing of the user's is in it+ shutil.rmtree(work, ignore_errors=True)+ raise+ added = len(w.entries) - len(keep)+ else:+ for f in orphans:+ os.remove(os.path.join(work, f))+ new_entries: list[dict] = []+ new_shards: list[dict] = []+ if dequants or fetch_srcs:+ log(f"in place: appending shard {_SHARD_FMT.format(next_num)} -> {work}")+ w = ShardWriter(work, index["shard_limit"], first_shard=next_num, global_off=index["total_bytes"])+ try:+ write_new(w)+ except BaseException:+ w.abort()+ raise+ new_shards = w.close()+ new_entries = w.entries+ next_num = w.shard_idx + 1+ tensors = [{**e, "name": name} for e, name in keep] + new_entries+ new_index = {**index, "tensors": tensors, "shards": index["shards"] + new_shards,+ "total_bytes": index["total_bytes"] + sum(sh["nbytes"] for sh in new_shards),+ "counts": {**index.get("counts", {}), "weight": sum(e["kind"] == "weight" for e in tensors)},+ "hotfix": hotfix}+ if renames or new_entries or drops:+ # rollback stays possible only while no shard gets replaced: keep the old index then+ bak = os.path.join(work, INDEX_NAME + ".bak")+ if not drops and not dirty and not os.path.exists(bak):+ shutil.copy2(os.path.join(work, INDEX_NAME), bak)+ write_index(work, new_index)+ log(f"index written: {os.path.join(work, INDEX_NAME)}")+ if need_ple:+ for f in ple_files:+ os.remove(os.path.join(work, f))+ got = ple_shards(work, source)+ side = {f for f in new_index.get("side_files", []) if not _PLE_FILE_RE.match(f)} | set(got)+ new_index = {**new_index, "side_files": sorted(side)}+ write_index(work, new_index)+ print(f" PLE table written: {len(got)} files")+ new_index, compacted = compact_in_place(work, new_index, next_num)+ if compacted:+ new_index = {**new_index, "hotfix": {**hotfix, "compacted": compacted}}+ write_index(work, new_index)+ # the old shards are gone, so an old index can no longer roll anything back+ bak = os.path.join(work, INDEX_NAME + ".bak")+ if os.path.exists(bak):+ os.remove(bak)+ added = len(new_entries)++ chk = check_ftw(work, new_index, expected, {}, set(), ple)+ ple_status, ple_detail, _ = chk["ple"]+ print(f"wrote {work}: {len(new_index['tensors'])} entries, +{added} new, {len(new_index['shards'])} shard(s)"+ + (f", {compacted} compacted" if compacted else "") + f", total {new_index['total_bytes'] / 2**30:.2f} GiB")+ print(f" check: missing {len(chk['missing'])} extra {len(chk['extra'])} problems {len(chk['problems'])} "+ f"dead bytes in {len(chk['dead'])} shard(s) PLE table: {ple_status}" + (f" ({ple_detail})" if ple_detail else ""))+ for n in (chk["missing"] + chk["extra"] + chk["problems"])[:10]:+ print(f" {n}")+ ok = not (chk["missing"] or chk["extra"] or chk["problems"] or chk["dead"]) and ple_status in ("complete", "n/a")+ return 0 if ok else 1+++if __name__ == "__main__":+ sys.exit(main())