FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
feat(scripts): add checkpoint diagnosis and fix scripts (#762)
Xiaoze Fan70550f8f9e36
diff --git a/AGENTS.md b/AGENTS.mdindex a2d263db..0d0878d9 100644--- a/AGENTS.md+++ b/AGENTS.md@@ -34,6 +34,8 @@ benchmarks/ end-to-end and micro benchmarks, see benchmarks/README.mddocs/ install, quickstart, CLI and model docsfreetoken-kernel-cache/ companion wheel of prebuilt kernels, see its READMEscripts/ wheel build and release scripts+ checkpoint_diagnose/ why an HF safetensors checkpoint does not load, and fixes for non-standard ones+ gguf_check/ which GGUF files of a family load```## Development@@ -54,6 +56,7 @@ Put a new test in the `tests/` directory that mirrors the module it protects, an- Search existing issues and PRs before starting. Items on the [Roadmap](https://github.com/FlashML-org/FreeToken/issues/79) are discussed with maintainers before implementation; features not on it start as an issue.- When helping the user draft an issue, follow the matching template in `.github/ISSUE_TEMPLATE/` (engine bug, model checkpoint, feature request) and fill in every required field: hardware, driver, FreeToken version, checkpoint ID, exact command, and the full log.+- When a checkpoint fails to load, follow [scripts/checkpoint_diagnose/README.md](scripts/checkpoint_diagnose/README.md) before drafting a model checkpoint issue: it tells a FreeToken gap from a non-standard checkpoint, and repairs the latter.- One change per PR, linked to its issue, with the hardware, checkpoint ID and exact command it was tested with.## Code commentsdiff --git a/README.md b/README.mdindex 6d9ab4ae..e72a5cd0 100644--- a/README.md+++ b/README.md@@ -56,6 +56,7 @@ For More details:- [Supported models](https://github.com/FlashML-org/FreeToken/blob/main/docs/models.md)- [CLI reference](https://github.com/FlashML-org/FreeToken/blob/main/docs/cli.md)- [Repairing old FTW checkpoints](https://github.com/FlashML-org/FreeToken/blob/main/docs/ftw-hotfix.md)+- [Diagnosing a checkpoint that fails to load](https://github.com/FlashML-org/FreeToken/blob/main/scripts/checkpoint_diagnose/README.md)## Citationdiff --git a/docs/models.md b/docs/models.mdindex 18f73fac..f6d99af9 100644--- a/docs/models.md+++ b/docs/models.md@@ -78,6 +78,7 @@ unset on PCIe GPUs: wider grids add no bandwidth there.## Notes+- A checkpoint that fails to load: from a source checkout, run `PYTHONPATH=python python scripts/checkpoint_diagnose/diagnose.py <checkpoint dir or HF repo id>`. It reads the headers, not the weights, says whether FreeToken or the checkpoint is at fault, and for a local checkpoint gives the commands that fix a non-standard one; see [its README](../scripts/checkpoint_diagnose/README.md). GGUF files are checked with [scripts/gguf_check](../scripts/gguf_check/README.md).- `ft checkpoint` conversion is optional — it pre-converts a checkpoint intoFreeToken's fast-load format, and `ft serve --model` auto-detects the result.- An FTW converted by an older build with `--moe-backend fused` or `triton` keepsdiff --git a/scripts/checkpoint_diagnose/README.md b/scripts/checkpoint_diagnose/README.mdnew file mode 100644index 00000000..57895288--- /dev/null+++ b/scripts/checkpoint_diagnose/README.md@@ -0,0 +1,177 @@+# checkpoint_diagnose++Find out why FreeToken does not load a checkpoint, sort the failure into one kind, and repair the checkpoints whose own files are non-standard.+Run everything from the repo root with `PYTHONPATH=python`. Nothing here needs a GPU or the checkpoint's weights.++> [!WARNING]+> The diagnosis is heuristic and can be wrong: the failure kind, the findings and the suggested fixes are a best guess from the headers and configs, not a proof. Check them against the problems it prints before acting on them.++> [!IMPORTANT]+> For agents: explain the diagnosis to the user in plain terms first: what fails, which kind it is, and what the fix would change. Ask the user before you run a fix script or open an issue. Do not change the user's checkpoint on your own.++| Script | Does |+|---|---|+| `diagnose.py` | checks a checkpoint dir or HF repo id, prints the failure kind and the next step |+| `fix_config.py` | level 1 fix: edits only `config.json` / `hf_quant_config.json` |+| `rename_tensors.py` | level 2 fix: renames tensors (shard headers, index, quant skip lists); tensor data is kept byte for byte |+| `fetch_headers.py` | header-only stub of an HF repo; `diagnose.py` calls it for a repo id |+| `ckpt_edit.py` | the edit primitives both fix scripts use, for a case they do not cover |++## Step 1: which format++`diagnose.py` routes by itself, but know where each format goes:++| The checkpoint has | Go to |+|---|---|+| `*.gguf` files | `scripts/gguf_check/` (its README), then the GGUF section below |+| `freetoken_weight.json` (FTW, made by `ft checkpoint`) | diagnose the HF checkpoint it was converted from; FTW repairs are in `docs/ftw-hotfix.md` |+| root `*.safetensors` shards | this directory, from step 2 |+| only `*.bin` / `*.pt`, or MLX weights | not served: FreeToken reads safetensors and GGUF |++### GGUF: three kinds of failure++`scripts/gguf_check/check.py` prints the problems of each file; the first one tells the kind.++| Kind | check.py says | Do |+|---|---|---|+| a ggml type has no kernel | `<layer kind> <type>: no <ops> kernel` (needs `--kernels` from `probe_kernels.py`) | the build or GPU lacks kernels for that quant type: pick a file with another type, or report it with the `probe_kernels.py` table |+| the model's loader does not read the file | `unexpected tensor ...`, `N tensors never loaded`, `tensor ... yielded twice`, `<tensor>: got ..., model has ...`, `MoE layer N ... bank is ..., the model needs ...`, or a `NotImplementedError` from the family's config parse | a FreeToken gap for that layout: report it with the check.py record |+| the GGUF file itself | `is not a GGUF file`, `duplicate GGUF key / tensor`, `runs past the end of the file`, `bad general.alignment`, `tokenizer: no EOS token`, `tokenizer: no chat template`, `chat template: <error>` | the file is broken or incomplete: download it again, or use another file of the repo |++A file whose architecture FreeToken does not serve shows as `skip`, not as a failure.++## Step 2: diagnose a safetensors checkpoint++```bash+python scripts/checkpoint_diagnose/diagnose.py <ckpt dir | HF repo id[@revision]> --out report.jsonl+```++- A local dir is read as is: the headers and JSON files, so a 400 GB checkpoint takes seconds. A weight reader that needs tensor values gets the real bytes it asks for.+- A repo id is fetched as a header-only stub into `~/.cache/freetoken/checkpoint-stubs/` (a few MB). Nothing is downloaded beyond that, and no `.py` file: a repo whose config is its own code (`auto_map`) is skipped, diagnose a local download of it instead.+- A remote repo is only diagnosed: its record has no fixes, and the fix scripts refuse a stub. When it says `fixable`, download the checkpoint and diagnose the local directory to get the fix commands.+- Add `--text-model-only` when the checkpoint will be served with `--text-model-only`.+- `--out` appends one JSON record per checkpoint. The fix scripts read their fixes from it with `--from-report`.++It first checks the files themselves (data offsets cover each shard exactly, the index agrees with the shard headers), then runs the server's own load path on the CPU with meta tensors: config parse, model build, dense weight load through the model's `load_state_dict`, and expert-bank fill for MoE models with the reader the engine picks.+When that fails, it also checks the tensor names against transformers' own model class and its checkpoint conversion rules, and lints the files for the known non-standard forms.++Output, per checkpoint: `PASS`, or `FAIL at <stage>: [<kind>] ...` with the problems, what transformers makes of the names, the findings, and the next step.++## Step 3: the kind of failure++Each failure gets one kind. `diagnose.py` tries, in this order: broken files (`checkpoint`), `arch`, `quant_method`, a broken prefix (`checkpoint`), `quant_model`, names transformers cannot map or another non-standard finding (`checkpoint`), `packing`; anything left is `loader`.++| Kind | Means | Do |+|---|---|---|+| `arch` | the architecture is not registered in `python/freetoken/models/register.py` | not served; a new architecture starts as a feature request |+| `quant_method` | FreeToken has no reader for the quantization format (GPTQ, AWQ, MLX, bitsandbytes, an unsupported compressed-tensors scheme, ...) | not served; use a release in a format `python/freetoken/layers/quantization/configs/` reads |+| `quant_model` | the format is read, but this model or layer kind has no method for it (`no quant method for moe.<kind>`), or the reader does not produce the scales the method needs | a FreeToken gap: report it |+| `loader` | the model and format are served, and transformers loads these names, but FreeToken's reader does not | a FreeToken gap: report it. A finding marked `loader` may offer a fix that works around it |+| `packing` | the expert banks do not fill, and transformers repacks these expert tensors (for example one tensor per expert where the family's release stacks them) | a FreeToken gap: report it. Repacking tensor data is not offered here |+| `checkpoint` | the checkpoint's own files are broken or non-standard: an incomplete shard or index, names transformers would not load either, or a config that contradicts what the shards store | fix it, step 4 |++To report a FreeToken gap, use the "model checkpoint" issue template and paste the `diagnose.py` output.++How to tell `loader` / `packing` from `checkpoint`: the line `transformers ... would not load it either` means transformers' model class and conversion rules cannot map the names either, so the checkpoint is at fault. `transformers loads these names` means the names are a form transformers accepts, so FreeToken should read them too.++## Step 4: fix a non-standard checkpoint++Fixes apply to a local copy of the checkpoint only, never to a remote repo or its stub.++Rules:++1. Fix into a new directory (`--out <new dir>`). Unchanged files there are symlinks to the original, so it costs only the rewritten shards. Use `--in-place` only when the user asks for it. It keeps a backup, and `--restore` undoes the latest in-place edit.+2. One level per round. If the record has a `rename` fix, apply that first, then run `diagnose.py` on the new directory, then apply the config fixes of that new record. A rename changes the names the config findings talk about.+3. Read the plan before applying it. Every run prints the plan first and changes nothing until it is confirmed. Check that:+ - each rule renames the tensors you expect, and the example `old -> new` lines are right;+ - nothing is renamed that should not be. A `warning: ... matches are inside the name` line means the pattern is not anchored;+ - the config edits name the right modules;+ - the shard costs are what you expect. `rewritten, copies N GiB` needs that much free disk.++ If anything in the plan is not what you expected, do not apply it: change the rule or the edit and plan again.+4. Confirming: in a terminal the script asks `[y/N]`. Without a terminal (an agent), pass `--plan <file>`. The first run saves the plan there and changes nothing. Read it, show it to the user, and run the exact same command again to apply it only after the user agrees. If the command or the checkpoint changed in between, the second run saves the new plan instead of applying anything.+5. After every fix, run `diagnose.py` on the result, until it passes or nothing fixable is left.++The usual round, for an agent:++```bash+D=scripts/checkpoint_diagnose+python $D/diagnose.py /models/x --out r0.jsonl+python $D/rename_tensors.py /models/x --out /models/x-fixed --from-report r0.jsonl --plan plan.json # prints and saves the plan+python $D/rename_tensors.py /models/x --out /models/x-fixed --from-report r0.jsonl --plan plan.json # after the user agrees to the plan: applies it+python $D/diagnose.py /models/x-fixed --out r1.jsonl+python $D/fix_config.py /models/x-fixed --in-place --from-report r1.jsonl --plan plan.json # twice, as above+python $D/diagnose.py /models/x-fixed+```++`--from-report` applies the fixes `diagnose.py` found for that checkpoint; anything else is a manual edit, below.++### Fix levels++| Level | Script | Changes | Typical use |+|---|---|---|---|+| 1 | `fix_config.py` | `config.json`, `hf_quant_config.json` | the quantization skip lists, compressed-tensors targets, a config key |+| 2 | `rename_tensors.py` | shard headers, `model.safetensors.index.json`, and the skip-list entries / targets that are module names or globs (a `re:` entry is not rewritten; the plan warns about every entry left naming the old modules) | broken prefixes, legacy tensor names |+| - | not offered | tensor data | repacking experts, dropping tensors, dequantizing: re-export the checkpoint, or report the FreeToken gap |++### Manual edits++When no automatic fix fits, write the edit yourself. It goes through the same plan and confirmation.++```bash+# rename: re.sub(PATTERN, REPL, name), rules in order; anchor with ^ and escape dots+python $D/rename_tensors.py /models/x --out /models/x-fixed --rule '^model\.language_model\.visual\.' 'model.visual.'++# config key: dotted path, JSON value+python $D/fix_config.py /models/x --out /models/x-fixed --set text_config.tie_word_embeddings false++# quantization skip lists, in the dialect's syntax:+# ModelOpt: glob (model.language_model.layers.*.mlp.gate)+# compressed-tensors: module name or re:<regex> (re:^model\.visual\..*)+# fp8: module name (model.language_model.layers.0.linear_attn.in_proj_a)+python $D/fix_config.py /models/x --out /models/x-fixed --skip-add 'model.language_model.layers.*.mlp.gate'+python $D/fix_config.py /models/x --out /models/x-fixed --skip-remove 'lm_head'+```++A rule that renames nothing, and a rename that gives two tensors the same name, are refused. So is an `--out` directory that already has files in it.++### Known non-standard forms++The findings `diagnose.py` prints, and what they fix. `kind` is the failure kind the finding points to.++| Finding | The checkpoint | Level | Fix |+|---|---|---|---|+| `prefix` | nests the text tower more than once (`model.language_model.language_model.`) | 2 | rename to `model.language_model.` |+| `prefix` | stores the vision tower under the text tower (`model.language_model.visual.`) | 2 | rename to `model.visual.` |+| `prefix` | uses vLLM's internal names (`language_model.model.`, `language_model.lm_head.`) | 2 | rename to `model.language_model.` / `lm_head.` |+| `quant_declared_but_dense` | stores modules in bf16 that its quantization_config does not skip (vision tower, lm_head) | 1 | add them to the skip lists |+| `quant_skipped_but_quantized` | stores modules quantized that its skip list covers, often through a container entry (`...linear_attn`) | 1 | remove the entry, and list its dense children instead |+| `ct_target_vllm_names` | writes compressed-tensors targets with vLLM's names, so they match no tensor | 1 | rewrite the targets to `model.language_model.` |+| `quant_unlisted_dense_router` (kind `loader`) | stores in bf16, without listing them as skipped, modules the family serves in bf16 anyway (the Qwen3.5 routers) | 1 | list them; works around the reader |+| `kv_output_scale` (kind `loader`) | stores fp8 KV scales as `k_proj.output_scale` / `v_proj.output_scale` (only reported for the families whose reader drops `k_scale`) | 2 | rename to `k_scale` / `v_scale`, which the reader drops |+| `expert_tensors_incomplete` | lacks some expert's tensors in some layer | - | download it again, or ask its uploader |+| (stage `files`) | has a shard whose tensors do not cover the file, or an index that disagrees with the shards | - | download it again |+| `no_vision_tower` | is text-only under a multimodal architecture | runtime | serve with `--text-model-only` |+| `no_processor_config` | has a vision tower but no `preprocessor_config.json` (image requests fail with 400) | file | copy that file from the base model |+| `fp8_without_config` | stores fp8 weights with no quantization_config | - | not fixable here; the scales and block size are unknown |+| `mlx` (kind `quant_method`) | is MLX-quantized (`.scales` / `.biases`) | - | not served; use a non-MLX release |++## Step 5: check the fixed checkpoint++`diagnose.py` passing means: the names, shapes, quantization methods and expert-bank layout all line up with what the model builds.+It does not read weight values. On a real checkpoint, finish with `ft serve --model-path <fixed dir>` and a short request. For a checkpoint with a vision tower, also send an image request.++## Using the primitives++```python+import sys; sys.path.insert(0, "scripts/checkpoint_diagnose")+from ckpt_edit import Checkpoint, confirm, plan_lines_for_save++ck = Checkpoint("/models/x")+ck.rename_regex([(r"^old\.", "new.")])+ck.quant_skip_add(["model.visual.*"])+print("\n".join(ck.log + plan_lines_for_save(ck, "/models/x-fixed")))+ck.save(out="/models/x-fixed") # ck.save() edits in place, with a backup+```++`ck.tensors()` gives every tensor's shard, dtype and shape from the headers. `ck.quant_configs()` gives every quantization config the checkpoint carries.diff --git a/scripts/checkpoint_diagnose/ckpt_edit.py b/scripts/checkpoint_diagnose/ckpt_edit.pynew file mode 100644index 00000000..5ec5215f--- /dev/null+++ b/scripts/checkpoint_diagnose/ckpt_edit.py@@ -0,0 +1,512 @@+"""Edit primitives for HF safetensors checkpoints: config keys, quantization skip lists, tensor names. Tensor data is never changed.++ from ckpt_edit import Checkpoint+ ck = Checkpoint("/path/to/ckpt")+ ck.set_config("text_config.tie_word_embeddings", False)+ ck.quant_skip_add(["model.language_model.layers.*.mlp.gate"])+ ck.rename(lambda name: name.replace("model.language_model.language_model.", "model.language_model."))+ ck.save(out="/path/to/fixed") # or ck.save() to edit in place++``save(out=...)`` writes a new directory: changed files are written there, every other file is a symlink to the original.+``save()`` edits in place and first copies each changed JSON file and shard header to ``.ckpt_edit_backup/<time>/``; ``restore()`` puts the latest backup back.+A renamed shard keeps its data bytes: its header is patched in place when the new one fits in the old one's bytes, else the shard is rewritten through a temp file.+A shard that is a symlink (hub cache) or has other hard links is never written through: it becomes a private copy.+"""++from __future__ import annotations++import copy+import errno+import fnmatch+import hashlib+import json+import os+import re+import shutil+import struct+import sys+import time+from typing import Any, Callable, Iterable++INDEX = "model.safetensors.index.json"+CONFIG = "config.json"+SIDECAR = "hf_quant_config.json"+BACKUP_DIR = ".ckpt_edit_backup"+STUB_MARKER = ".stub.json"+# keys that list modules a quantization_config leaves unquantized+SKIP_KEYS = ("ignore", "exclude_modules", "modules_to_not_convert")+_DEFAULT_SKIP_KEY = {"modelopt": "ignore", "compressed-tensors": "ignore", "fp8": "modules_to_not_convert", "mxfp4": "modules_to_not_convert"}+++def header_len(path: str) -> int:+ with open(path, "rb") as f:+ return struct.unpack("<Q", f.read(8))[0]+++def read_header(path: str) -> tuple[dict, int]:+ """(header, header byte length) of a safetensors file."""+ with open(path, "rb") as f:+ (n,) = struct.unpack("<Q", f.read(8))+ return json.loads(f.read(n)), n+++def encode_header(header: dict, fit: int | None = None) -> bytes:+ """The header JSON padded with spaces to ``fit`` bytes when it fits, else to a multiple of 8."""+ raw = json.dumps(header, separators=(",", ":")).encode()+ size = fit if fit is not None and len(raw) <= fit else (len(raw) + 7) // 8 * 8+ return raw + b" " * (size - len(raw))+++def _copy_range(src_fd: int, dst_fd: int, src_off: int, dst_off: int, count: int) -> None:+ fast = hasattr(os, "copy_file_range")+ while count:+ n = 0+ if fast:+ try:+ n = os.copy_file_range(src_fd, dst_fd, count, src_off, dst_off)+ except OSError:+ fast = False+ if n <= 0:+ buf = os.pread(src_fd, min(count, 64 << 20), src_off)+ if not buf:+ raise EOFError("source shard ended early")+ n = os.pwrite(dst_fd, buf, dst_off)+ count, src_off, dst_off = count - n, src_off + n, dst_off + n+++def stub_data_end(path: str, n: int) -> int | None:+ """For a fetch_headers.py stub, the end of the block that holds the header; past it the file is a hole. None for anything else.++ A shard counts as a stub only next to the stub marker and when SEEK_DATA finds no data past the header's block."""+ if not os.path.isfile(os.path.join(os.path.dirname(os.path.abspath(path)), STUB_MARKER)):+ return None+ st = os.stat(path)+ end = min(st.st_size, (8 + n + st.st_blksize - 1) // st.st_blksize * st.st_blksize)+ if end >= st.st_size:+ return None+ fd = os.open(path, os.O_RDONLY)+ try:+ os.lseek(fd, end, os.SEEK_DATA)+ except OSError as e:+ return end if e.errno == errno.ENXIO else None+ finally:+ os.close(fd)+ return None+++def _fsync_dir(path: str) -> None:+ fd = os.open(os.path.dirname(os.path.abspath(path)) or ".", os.O_RDONLY)+ try:+ os.fsync(fd)+ finally:+ os.close(fd)+++def write_shard(src: str, dst: str, header: dict | bytes, *, private: bool = False) -> str:+ """Write ``header`` (a dict, or the exact header bytes) with the data bytes of ``src`` to ``dst``; returns ``"in place"`` or ``"rewritten"``.++ ``private`` forces a rewrite through a temp file even when ``dst`` is ``src``, so a symlink or a hard link is replaced instead of written through."""+ n_old = header_len(src)+ size = os.path.getsize(src)+ same = os.path.realpath(src) == os.path.realpath(dst)+ data = header if isinstance(header, bytes) else encode_header(header, fit=n_old if same else None)+ if same and not private and len(data) == n_old:+ fd = os.open(dst, os.O_WRONLY)+ try:+ os.pwrite(fd, data, 8)+ os.fsync(fd)+ finally:+ os.close(fd)+ return "in place"+ tmp = dst + ".ckpt_edit.part"+ head = struct.pack("<Q", len(data)) + data+ stub_end = stub_data_end(src, n_old)+ src_fd, dst_fd = os.open(src, os.O_RDONLY), os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o644)+ try:+ os.pwrite(dst_fd, head, 0)+ if stub_end is not None:+ # a stub's data is a hole: copy the bytes up to the end of the header's block, keep the rest a hole+ _copy_range(src_fd, dst_fd, 8 + n_old, len(head), stub_end - 8 - n_old)+ os.ftruncate(dst_fd, len(head) + size - 8 - n_old)+ else:+ _copy_range(src_fd, dst_fd, 8 + n_old, len(head), size - 8 - n_old)+ os.fsync(dst_fd)+ except BaseException:+ os.close(dst_fd)+ os.remove(tmp)+ raise+ finally:+ os.close(src_fd)+ os.close(dst_fd)+ os.replace(tmp, dst)+ _fsync_dir(dst)+ return "rewritten"+++def _shared(path: str) -> bool:+ """A symlink or a file with other hard links: writing through it would change another copy."""+ return os.path.islink(path) or os.stat(path).st_nlink > 1+++def get_path(cfg: dict, path: str) -> Any:+ for key in path.split("."):+ if not isinstance(cfg, dict) or key not in cfg:+ return None+ cfg = cfg[key]+ return cfg+++def set_path(cfg: dict, path: str, value: Any) -> None:+ *parents, leaf = path.split(".")+ for key in parents:+ cfg = cfg.setdefault(key, {})+ cfg[leaf] = value+++def entry_names(entry: str, module: str) -> bool:+ """Whether a skip-list entry or target covers ``module``, read loosely across dialects (literal name or parent, glob, ``re:``)."""+ if entry.startswith("re:"):+ try:+ return re.match(entry[3:], module) is not None+ except re.error:+ return False+ names = [module]+ while "." in names[-1]:+ names.append(names[-1].rpartition(".")[0])+ return any(n == entry or fnmatch.fnmatchcase(n, entry) for n in names)+++class Checkpoint:+ def __init__(self, path: str):+ self.path = os.path.abspath(path)+ self.json: dict[str, dict] = {}+ for name in (CONFIG, SIDECAR, INDEX):+ p = os.path.join(self.path, name)+ if os.path.isfile(p):+ with open(p, encoding="utf-8") as f:+ self.json[name] = json.load(f)+ self._orig_json = copy.deepcopy(self.json)+ self.headers: dict[str, dict] = {}+ self.header_len: dict[str, int] = {}+ digest = hashlib.sha256(json.dumps(self.json, sort_keys=True).encode())+ for name in sorted(os.listdir(self.path)):+ if name.endswith(".safetensors"):+ self.headers[name], self.header_len[name] = read_header(os.path.join(self.path, name))+ digest.update(name.encode() + json.dumps(self.headers[name], sort_keys=True).encode())+ # the state the edits are planned against; a plan made on another state is refused+ self.fingerprint = digest.hexdigest()+ self._changed_shards: set[str] = set()+ self.log: list[str] = []++ # --- reading ---++ def tensors(self) -> dict[str, tuple[str, dict]]:+ """name -> (shard, {dtype, shape, data_offsets})."""+ return {k: (shard, v) for shard, h in self.headers.items() for k, v in h.items() if k != "__metadata__"}++ @property+ def config(self) -> dict:+ return self.json.setdefault(CONFIG, {})++ def quant_configs(self) -> list[tuple[str, dict]]:+ """(where, dict) of every quantization config the checkpoint carries: config.json top level, its text_config, and hf_quant_config.json."""+ found = []+ for where, q in ((CONFIG, get_path(self.config, "quantization_config")),+ (f"{CONFIG}:text_config", get_path(self.config, "text_config.quantization_config")),+ (SIDECAR, get_path(self.json.get(SIDECAR, {}), "quantization"))):+ if isinstance(q, dict):+ found.append((where, q))+ return found++ def _method(self, where: str, q: dict) -> str:+ return "modelopt" if where == SIDECAR else str(q.get("quant_method") or "").lower()++ def quant_entries(self) -> list[tuple[str, str, list]]:+ """(where, key, list) of every skip list and compressed-tensors targets list."""+ lists = []+ for where, q in self.quant_configs():+ lists += [(where, k, q[k]) for k in SKIP_KEYS if isinstance(q.get(k), list)]+ for gname, group in (q.get("config_groups") or {}).items():+ if isinstance(group, dict) and isinstance(group.get("targets"), list):+ lists.append((where, f"config_groups.{gname}.targets", group["targets"]))+ return lists++ # --- config edits ---++ def set_config(self, path: str, value: Any) -> None:+ old = get_path(self.config, path)+ set_path(self.config, path, value)+ self.log.append(f"config.json {path}: {json.dumps(old)} -> {json.dumps(value)}")++ def quant_skip_add(self, entries: Iterable[str]) -> None:+ """Add module names or patterns to the skip lists of every quantization config, creating the dialect's list if it has none.++ An empty list next to a non-empty one is left alone (ModelOpt readers take ``ignore`` only when it is non-empty); an fp8 list that is created or empty first gets the ``lm_head`` the dialect skips by default."""+ entries = list(entries)+ for where, q in self.quant_configs():+ method = self._method(where, q)+ keys = [k for k in SKIP_KEYS if isinstance(q.get(k), list)]+ if not keys:+ keys = ["exclude_modules" if where == SIDECAR else _DEFAULT_SKIP_KEY.get(method, "ignore")]+ q[keys[0]] = []+ if any(q[k] for k in keys):+ keys = [k for k in keys if q[k]]+ for key in keys:+ lst = q[key]+ implicit = ["lm_head"] if method == "fp8" and key == "modules_to_not_convert" and not lst else []+ new = list(dict.fromkeys(e for e in implicit + entries if e not in lst))+ lst.extend(new)+ if new:+ self.log.append(f"{where} {key}: added {new}")++ def quant_skip_remove(self, entries: Iterable[str]) -> None:+ entries = set(entries)+ for where, q in self.quant_configs():+ for key in SKIP_KEYS:+ lst = q.get(key)+ if isinstance(lst, list):+ gone = [e for e in lst if e in entries]+ lst[:] = [e for e in lst if e not in entries]+ if gone:+ self.log.append(f"{where} {key}: removed {gone}")++ def quant_sub(self, pattern: str, repl: str) -> None:+ """re.sub over the text of every skip-list entry and compressed-tensors target, ``re:`` entries included."""+ self._sub_entries(lambda e: re.sub(pattern, repl, e))++ def quant_rename(self, pattern: str, repl: str) -> None:+ """Apply a tensor rename rule to the skip-list entries and targets that are module names or globs; ``re:`` entries are left alone.++ Each entry is matched as ``entry + "."``, so a rule written for tensor names also renames a container entry."""+ rx = re.compile(pattern)+ self._sub_entries(lambda e: e if e.startswith("re:") else rx.sub(repl, e + ".").removesuffix("."))++ def _sub_entries(self, fn: Callable[[str], str]) -> None:+ for where, key, lst in self.quant_entries():+ new = [fn(e) for e in lst]+ if new != lst:+ self.log.append(f"{where} {key}: {[e for e in lst if e not in new]} -> {[e for e in new if e not in lst]}")+ lst[:] = new++ def stale_quant_entries(self, old_modules: Iterable[str], new_modules: Iterable[str]) -> list[str]:+ """Skip-list entries and targets that named some of ``old_modules`` and name none of ``new_modules``."""+ old, new = list(old_modules), list(new_modules)+ stale = []+ for where, key, lst in self.quant_entries():+ for e in lst:+ if e != "Linear" and any(entry_names(e, m) for m in old) and not any(entry_names(e, m) for m in new):+ stale.append(f"{where} {key}: {e!r}")+ return stale++ # --- tensor names ---++ def rename(self, fn: Callable[[str], str]) -> dict[str, str]:+ """Rename tensors by ``fn(name)``; returns {old: new} of the names that changed. Refuses a rename that makes two tensors share a name."""+ changes = {}+ for shard, h in self.headers.items():+ for k in h:+ if k != "__metadata__" and fn(k) != k:+ changes[k] = (shard, fn(k))+ if not changes:+ return {}+ final = [changes[k][1] if k in changes else k for k in self.tensors()]+ if len(final) != len(set(final)):+ dup = sorted({n for n in final if final.count(n) > 1})[:5]+ raise ValueError(f"the rename gives two tensors the same name, e.g. {dup}")+ for old, (shard, new) in changes.items():+ h = self.headers[shard]+ h[new] = h.pop(old)+ self._changed_shards.add(shard)+ index = self.json.get(INDEX)+ if index is not None:+ wm = index["weight_map"]+ index["weight_map"] = {(changes[k][1] if k in changes else k): v for k, v in wm.items()}+ self.log.append(f"renamed {len(changes)} tensors in {len({s for s, _ in changes.values()})} shards")+ return {old: new for old, (_, new) in changes.items()}++ def rename_regex(self, rules: Iterable[tuple[str, str]]) -> dict[str, str]:+ """Apply ``re.sub(pattern, repl, name)`` for each rule in order."""+ compiled = [(re.compile(p), r) for p, r in rules]++ def fn(name: str) -> str:+ for rx, repl in compiled:+ name = rx.sub(repl, name)+ return name++ return self.rename(fn)++ # --- writing ---++ def _changed_json(self) -> list[str]:+ return [n for n, v in self.json.items() if v != self._orig_json.get(n)]++ def pending(self, out: str | None) -> dict:+ """What ``save(out)`` would write: {"json": [file names], "shards": {shard: how}}."""+ shards = {}+ for shard in sorted(self._changed_shards):+ p, n = os.path.join(self.path, shard), self.header_len[shard]+ fits = len(encode_header(self.headers[shard], fit=n)) == n+ if out is None and fits and not _shared(p):+ shards[shard] = "header patched in place"+ elif stub_data_end(p, n) is not None:+ shards[shard] = "rewritten (header-only stub, no data copied)"+ else:+ why = " (a shared symlink / hard link becomes a private copy)" if out is None and _shared(p) else ""+ shards[shard] = f"rewritten, copies {os.path.getsize(p) / 2**30:.1f} GiB{why}"+ return {"json": self._changed_json(), "shards": shards}++ def save(self, out: str | None = None) -> list[str]:+ """Write the edits; returns what was written. ``out=None`` edits in place (with a backup)."""+ changed_json, written = self._changed_json(), []+ if out is None:+ if not changed_json and not self._changed_shards:+ return []+ backup = os.path.join(self.path, BACKUP_DIR, time.strftime("%Y%m%d-%H%M%S") + f"-{time.time_ns() % 10**9:09d}")+ os.makedirs(backup)+ manifest = {"links": {}, "created": []}+ for name in changed_json:+ p = os.path.join(self.path, name)+ if os.path.lexists(p):+ if os.path.islink(p):+ manifest["links"][name] = os.readlink(p)+ shutil.copyfile(p, os.path.join(backup, name))+ else:+ manifest["created"].append(name)+ for shard in self._changed_shards:+ p = os.path.join(self.path, shard)+ if os.path.islink(p):+ manifest["links"][shard] = os.readlink(p)+ with open(p, "rb") as f:+ (n,) = struct.unpack("<Q", f.read(8))+ head = f.read(n)+ with open(os.path.join(backup, shard + ".header"), "wb") as f:+ f.write(head)+ with open(os.path.join(backup, "manifest.json"), "w") as f:+ json.dump(manifest, f)+ for shard in sorted(self._changed_shards):+ p = os.path.join(self.path, shard)+ written.append(f"{shard} ({write_shard(p, p, self.headers[shard], private=_shared(p))})")+ target = self.path+ else:+ target = os.path.realpath(out)+ if target == os.path.realpath(self.path) or target.startswith(os.path.realpath(self.path) + os.sep):+ raise ValueError("--out must be outside the checkpoint directory")+ os.makedirs(target, exist_ok=True)+ for name in sorted(os.listdir(self.path)):+ src, dst = os.path.join(self.path, name), os.path.join(target, name)+ if name == BACKUP_DIR or name in changed_json or name in self._changed_shards or os.path.lexists(dst):+ continue+ os.symlink(os.path.realpath(src), dst)+ for shard in sorted(self._changed_shards):+ write_shard(os.path.join(self.path, shard), os.path.join(target, shard), self.headers[shard])+ written.append(f"{shard} (rewritten)")+ for name in changed_json:+ dst = os.path.join(target, name)+ tmp = dst + ".ckpt_edit.part"+ with open(tmp, "w", encoding="utf-8") as f:+ json.dump(self.json[name], f, indent=2, ensure_ascii=False)+ f.write("\n")+ f.flush()+ os.fsync(f.fileno())+ # replaces a symlink without writing through it+ os.replace(tmp, dst)+ written.append(name)+ self._orig_json = copy.deepcopy(self.json)+ self._changed_shards.clear()+ return written+++def confirm(plan: dict, lines: list[str], plan_file: str | None) -> bool:+ """Print the plan, then decide whether to apply it.++ With ``plan_file`` (no prompt): a run whose plan is not already in the file writes it there and changes nothing; the next run of the same command applies it only if it computes the same plan from the same checkpoint state.+ Without: ask [y/N], and change nothing when stdin is not a terminal."""+ print("\n".join(lines))+ if plan_file:+ saved = None+ if os.path.exists(plan_file):+ try:+ with open(plan_file) as f:+ saved = json.load(f)+ except (OSError, ValueError):+ saved = None+ if not isinstance(saved, dict) or "fingerprint" not in saved:+ raise SystemExit(f"--plan {plan_file} exists and is not a plan saved by these scripts; pick another path")+ if saved == plan:+ print(f"\nthis is the plan saved in {plan_file}: applying it")+ return True+ with open(plan_file, "w") as f:+ json.dump(plan, f, indent=1)+ what = "differs from the plan saved there (the command or the checkpoint changed); saved the new one" if saved else "saved"+ print(f"\nplan {what} in {plan_file}. Nothing was changed: check the plan, then run the same command again to apply it.")+ return False+ if not sys.stdin.isatty():+ print("\nstdin is not a terminal, so nothing was changed. Pass --plan <file>: the first run saves the plan, a second identical run applies it.")+ return False+ return input("\napply this plan? [y/N] ").strip().lower() in ("y", "yes")+++def plan_lines_for_save(ck: "Checkpoint", out: str | None) -> list[str]:+ todo = ck.pending(out)+ lines = [f" write: {', '.join(todo['json']) or 'no JSON file'}"]+ lines += [f" shard {s}: {how}" for s, how in todo["shards"].items()]+ if out is None:+ lines.append(f" in place: the originals of these files go to {os.path.join(ck.path, BACKUP_DIR)}/ first (--restore puts them back)")+ else:+ lines.append(f" into {os.path.realpath(out)}: every other file is a symlink to the original")+ return lines+++def refuse_stub(path: str) -> None:+ """The fix scripts edit real checkpoints only: a fetch_headers.py stub of a remote repo is for diagnosis."""+ if os.path.isfile(os.path.join(path, STUB_MARKER)):+ raise SystemExit(f"{path} is a header-only stub of an HF repo, which is only diagnosed: download the checkpoint, run diagnose.py on the local directory, and fix that")+++def check_out_dir(ck: "Checkpoint", out: str | None) -> None:+ if out is None:+ return+ target, root = os.path.realpath(out), os.path.realpath(ck.path)+ if target == root or target.startswith(root + os.sep):+ raise SystemExit("--out must be outside the checkpoint directory")+ if os.path.isdir(target) and os.listdir(target):+ raise SystemExit(f"--out {target} exists and is not empty; pick a new directory")+++def restore(path: str) -> list[str]:+ """Put back the latest in-place backup of ``path``; returns what was restored."""+ root = os.path.join(path, BACKUP_DIR)+ stamps = sorted(os.listdir(root)) if os.path.isdir(root) else []+ if not stamps:+ raise FileNotFoundError(f"no backup under {root}")+ backup, done = os.path.join(root, stamps[-1]), []+ manifest_path = os.path.join(backup, "manifest.json")+ manifest = {"links": {}, "created": []}+ if os.path.isfile(manifest_path):+ with open(manifest_path) as f:+ manifest = json.load(f)+ for name in sorted(os.listdir(backup)):+ if name == "manifest.json":+ continue+ target = name.removesuffix(".header")+ p = os.path.join(path, target)+ if target in manifest["links"]:+ # the edit made a private copy of a shared file: point the name back at the original+ tmp = p + ".ckpt_edit.link"+ os.symlink(manifest["links"][target], tmp)+ os.replace(tmp, p)+ elif name.endswith(".safetensors.header"):+ if os.path.islink(p):+ raise SystemExit(f"{p} is a symlink now; the backup was taken of a regular file, restore it by hand")+ with open(os.path.join(backup, name), "rb") as f:+ write_shard(p, p, f.read())+ else:+ shutil.copyfile(os.path.join(backup, name), p + ".ckpt_edit.part")+ os.replace(p + ".ckpt_edit.part", p)+ done.append(target)+ for name in manifest["created"]:+ if os.path.exists(os.path.join(path, name)):+ os.remove(os.path.join(path, name))+ done.append(f"{name} (removed, the edit created it)")+ shutil.rmtree(backup)+ return donediff --git a/scripts/checkpoint_diagnose/diagnose.py b/scripts/checkpoint_diagnose/diagnose.pynew file mode 100644index 00000000..0b1b1e44--- /dev/null+++ b/scripts/checkpoint_diagnose/diagnose.py@@ -0,0 +1,1040 @@+"""Diagnose why FreeToken does not load an HF safetensors checkpoint, without reading its weights.++ python scripts/checkpoint_diagnose/diagnose.py <ckpt dir or HF repo[@revision]> ... [--out report.jsonl] [--text-model-only]++A repo id is fetched as a header-only stub (``fetch_headers.py``) into ~/.cache/freetoken/checkpoint-stubs first. GGUF and FTW checkpoints are pointed elsewhere.+Per checkpoint, through the code the server runs, on the CPU and the meta device:++- config: the architecture is registered and the quantization_config parses+- build: the model builds with a quant method for every layer+- dense load: the family's weight reader yields exactly the model's state dict (names and shapes)+- experts: every MoE layer's expert banks fill, packed by the layer's expert method++The readers get meta tensors with the header's shapes and dtypes; a reader that needs values gets a second run on the real bytes (zeros in a stub).+On a failure it also checks the tensor names against transformers' own model and its checkpoint conversion rules, and lints the files for the known non-standard forms.+The failure is then put in one category (see README.md), with the fixes the fix scripts can apply; ``--out`` keeps them for ``--from-report``.+"""++from __future__ import annotations++import argparse+import concurrent.futures as cf+import contextlib+import functools+import json+import logging+import multiprocessing as mp+import os+import re+import struct+import sys+import time+import traceback+import warnings+from collections import Counter++HERE = os.path.dirname(os.path.abspath(__file__))+sys.path.insert(0, HERE)+STUB_DIR = os.path.expanduser("~/.cache/freetoken/checkpoint-stubs")++CATEGORIES = {+ "arch": "architecture not served",+ "quant_method": "quantization method not supported",+ "quant_model": "quantization method supported, but not for this model or layer",+ "loader": "model supported, the loader does not read this checkpoint",+ "packing": "the checkpoint packs tensors in a form the loader does not read",+ "checkpoint": "the checkpoint itself is non-standard",+}+FIX_SCRIPTS = {"config": "fix_config.py", "rename": "rename_tensors.py"}+# tensors stored next to a quantized weight; they fold into the module's .weight when the names are compared with transformers+QUANT_COMPANIONS = (+ "weight_scale", "weight_scale_2", "weight_scale_inv", "weight_global_scale", "input_scale", "input_global_scale",+ "weight_zero_point", "k_scale", "v_scale", "q_scale", "prob_scale", "output_scale", "weight_shape",+)+_QUANT_DTYPES = ("U8", "I8", "F8_E4M3", "F8_E5M2", "I32", "F4")+_DIGITS = re.compile(r"\.\d+(?=\.|$)")+# known non-standard prefixes: (pattern, replacement, what is wrong, checkpoint root the family expects)+PREFIX_RULES = (+ (r"^model\.language_model\.(?:language_model\.)+", "model.language_model.", "the text tower is nested more than once under language_model", "model.language_model"),+ (r"^model\.language_model\.visual\.", "model.visual.", "the vision tower is stored under the text tower", "model.visual"),+ (r"^language_model\.model\.", "model.language_model.", "the text tower uses the older language_model.model. layout", "model.language_model"),+ (r"^language_model\.lm_head\.", "lm_head.", "lm_head uses the older language_model.lm_head layout", "model.language_model"),+)+# fp8 KV calibration scales under the projection's name; vLLM reads them as attn.k_scale, which FreeToken drops+KV_SCALE_RULE = (r"\.([kv])_proj\.output_scale$", r".\1_scale")+# the families whose weight reader drops .k_scale / .v_scale tensors; for the others the rename only moves the unexpected key+KV_SCALE_DROPPERS = ("freetoken.models.qwen3_5_moe",)+_PER_EXPERT = re.compile(r"^(?P<pre>.*\.layers\.)(?P<layer>\d+)(?P<mid>\..*\.experts\.)(?P<e>\d+)\.(?P<proj>[^.]+)\.")+# compressed-tensors targets written with vLLM's internal module names+TARGET_SWAPS = ((r"language_model\[\.\]model\[\.\]", "model[.]language_model[.]"), (r"language_model\\\.model\\\.", r"model\\.language_model\\."))+# per-module tensors a quant method's reader produces next to the weight+_SCALE_ROLES = (".weight_scale", ".weight_scale_inv", ".weight_scale_2", ".weight_global", ".input_scale")+++def pattern_of(name: str) -> str:+ return _DIGITS.sub(".N", name)+++def group_names(names, limit: int = 4) -> str:+ """Names grouped by their pattern (layer and expert numbers as N), most common first."""+ c = Counter(pattern_of(n) for n in names)+ parts = [f"{p} x{n}" if n > 1 else p for p, n in c.most_common(limit)]+ return ", ".join(parts) + (f", ... ({len(c) - limit} more patterns)" if len(c) > limit else "")+++def shard_files(path: str) -> list[str]:+ """The root shards the loaders read: consolidated.safetensors only when it is the only one (loader.iter_weight_files)."""+ files = sorted(f for f in os.listdir(path) if f.endswith(".safetensors"))+ return [f for f in files if f != "consolidated.safetensors"] or files+++def read_tensor_table(path: str) -> dict[str, dict]:+ """name -> {dtype, shape, shard} from every shard header."""+ table = {}+ for f in shard_files(path):+ for k, v in _header(os.path.join(path, f)).items():+ if k != "__metadata__":+ table[k] = {"dtype": v["dtype"], "shape": v["shape"], "shard": f}+ return table+++_ITEM_BITS = {"F64": 64, "I64": 64, "U64": 64, "F32": 32, "I32": 32, "U32": 32, "F16": 16, "BF16": 16, "I16": 16, "U16": 16,+ "I8": 8, "U8": 8, "BOOL": 8, "F8_E4M3": 8, "F8_E5M2": 8, "F8_E8M0": 8, "F4": 4}+++def check_files(path: str) -> list[str]:+ """What safetensors and the loaders would refuse in the files themselves: data offsets that leave gaps or do not cover the file, a tensor whose bytes do not match its shape, an index that disagrees with the shard headers."""+ problems, names = [], {}+ for f in shard_files(path):+ p = os.path.join(path, f)+ with open(p, "rb") as fh:+ (n,) = struct.unpack("<Q", fh.read(8))+ header = _header(p)+ spans = sorted((v["data_offsets"][0], v["data_offsets"][1], k) for k, v in header.items() if k != "__metadata__")+ pos = 0+ for start, end, k in spans:+ if start != pos:+ problems.append(f"{f}: tensor {k} starts at byte {start} of the data, expected {pos} (gap or overlap)")+ break+ bits = _ITEM_BITS.get(header[k]["dtype"])+ numel = 1+ for d in header[k]["shape"]:+ numel *= d+ if bits is not None and end - start != numel * bits // 8:+ problems.append(f"{f}: tensor {k} has {end - start} bytes, its shape {header[k]['shape']} {header[k]['dtype']} needs {numel * bits // 8}")+ break+ pos = end+ size = os.path.getsize(p) - 8 - n+ if not problems and pos != size:+ problems.append(f"{f}: the tensors cover {pos} data bytes but the file has {size} (an incomplete download?)")+ names.update({k: f for k in header if k != "__metadata__"})+ index = os.path.join(path, "model.safetensors.index.json")+ if os.path.isfile(index):+ with open(index) as fh:+ wm = json.load(fh).get("weight_map", {})+ read = set(shard_files(path))+ # entries for shards the loaders skip (a subdir, an unread consolidated.safetensors) say nothing about the read ones+ wm = {k: v for k, v in wm.items() if "/" not in v and (v in read or v != "consolidated.safetensors")}+ absent = [k for k in wm if k not in names]+ unlisted = [k for k in names if k not in wm]+ moved = [k for k in wm if k in names and wm[k] != names[k]]+ if absent:+ problems.append(f"the index lists {len(absent)} tensors no shard holds: {group_names(absent)}")+ if unlisted:+ problems.append(f"the shards hold {len(unlisted)} tensors the index does not list: {group_names(unlisted)}")+ if moved:+ problems.append(f"the index puts {len(moved)} tensors in another shard than the one that holds them: {group_names(moved)}")+ return problems+++def split_module(name: str) -> tuple[str, str]:+ """(module, leaf) of a tensor name: model.layers.0.mlp.gate.weight_scale -> (model.layers.0.mlp.gate, weight_scale)."""+ module, _, leaf = name.rpartition(".")+ return module, leaf+++def stored_modules(table: dict) -> dict[str, tuple[bool, str]]:+ """module -> (stored quantized, weight dtype) for every module with a 2-D weight or a packed weight."""+ out = {}+ by_module: dict[str, dict[str, dict]] = {}+ for name, meta in table.items():+ module, leaf = split_module(name)+ by_module.setdefault(module, {})[leaf] = meta+ for module, leaves in by_module.items():+ w = leaves.get("weight_packed") or leaves.get("weight")+ if w is None or (len(w["shape"]) != 2 and "weight_packed" not in leaves):+ continue+ quantized = "weight_packed" in leaves or w["dtype"] in _QUANT_DTYPES+ out[module] = (quantized, w["dtype"])+ return out+++# --- format routing (main process, no torch) ---+++def route(path: str) -> dict | None:+ """A record for a checkpoint this script does not check (GGUF, FTW, no safetensors), else None."""+ files = os.listdir(path)+ if "freetoken_weight.json" in files:+ return {"path": path, "status": "skip", "format": "ftw",+ "hint": "an FTW checkpoint converted by `ft checkpoint`: diagnose its source HF checkpoint; docs/ftw-hotfix.md covers FTW repairs"}+ stub = _stub_record(path)+ gguf = any(f.endswith(".gguf") for f in files) or bool(stub and stub.get("gguf"))+ shards = [f for f in files if f.endswith(".safetensors")]+ if gguf and not shards:+ return {"path": path, "status": "skip", "format": "gguf",+ "hint": "a GGUF checkpoint: check it with scripts/gguf_check/ (README.md there); the GGUF section of this README sorts its failures"}+ code = _missing_config_code(path) if stub else []+ if code:+ return {"path": path, "status": "skip", "format": "remote code",+ "hint": f"the repo ships its own config code ({', '.join(code)}), which a header-only stub does not fetch and FreeToken runs with trust_remote_code; download the repo and diagnose the local directory"}+ if not shards:+ other = sorted({os.path.splitext(f)[1] for f in files if f.endswith((".bin", ".pt", ".pth", ".npz"))} | set((stub or {}).get("other_weights") or ()))+ return {"path": path, "status": "skip", "format": "other",+ "hint": f"no root *.safetensors shard ({', '.join(other) or 'no weights found'}); FreeToken reads safetensors and GGUF only"}+ return None+++def _missing_config_code(path: str) -> list[str]:+ """The .py files the config's auto_map names for AutoConfig that are not in ``path``."""+ try:+ with open(os.path.join(path, "config.json")) as f:+ cfg = json.load(f)+ except (OSError, ValueError):+ return []+ files = []+ for c in (cfg, cfg.get("text_config") or {}):+ ref = (c.get("auto_map") or {}).get("AutoConfig") if isinstance(c, dict) else None+ if isinstance(ref, str) and "--" not in ref:+ files.append(ref.rsplit(".", 1)[0] + ".py")+ return [f for f in dict.fromkeys(files) if not os.path.isfile(os.path.join(path, f))]+++def _stub_record(path: str) -> dict | None:+ p = os.path.join(path, ".stub.json")+ if not os.path.isfile(p):+ return None+ with open(p) as f:+ return json.load(f)+++# --- meta-device readers (worker) ---+++def _header(path: str) -> dict:+ with open(path, "rb") as f:+ (n,) = struct.unpack("<Q", f.read(8))+ return json.loads(f.read(n))+++def _meta_tensor(meta: dict):+ import torch+ from freetoken.models.weight import _ST_DTYPE++ return torch.empty(meta["shape"], dtype=_ST_DTYPE[meta["dtype"]], device="meta")+++class _MetaSlice:+ def __init__(self, meta: dict):+ self.meta = meta++ def get_shape(self):+ return list(self.meta["shape"])++ def get_dtype(self):+ return self.meta["dtype"]++ def __getitem__(self, idx):+ return _meta_tensor(self.meta)[idx]+++class _MetaSafeOpen:+ """safetensors.safe_open whose tensors are meta tensors with the header's shape and dtype."""++ def __init__(self, filename, framework="pt", device="cpu"):+ self._h = _header(os.fspath(filename))++ def __enter__(self):+ return self++ def __exit__(self, *exc):+ return False++ def keys(self):+ return [k for k in self._h if k != "__metadata__"]++ def offset_keys(self):+ return self.keys()++ def metadata(self):+ return self._h.get("__metadata__")++ def get_tensor(self, name):+ return _meta_tensor(self._h[name])++ def get_slice(self, name):+ return _MetaSlice(self._h[name])+++def _meta_iter_expert_tensors_parallel(model_path, is_expert, **_kwargs):+ from freetoken.models.loader import safetensors_weight_map+ from freetoken.utils.hf import download_hf_weight++ folder = download_hf_weight(model_path)+ for name, shard in safetensors_weight_map(folder).items():+ if is_expert(name):+ yield name, _meta_tensor(_header(os.path.join(folder, shard))[name])+++def _meta_wins():+ """A torch function mode that moves CPU operands to meta when an op mixes the two."""+ import torch+ from torch.overrides import TorchFunctionMode+ from torch.utils._pytree import tree_flatten, tree_unflatten++ class MetaWins(TorchFunctionMode):+ def __torch_function__(self, func, types, args=(), kwargs=None):+ flat, spec = tree_flatten((args, kwargs or {}))+ if any(isinstance(x, torch.Tensor) and x.is_meta for x in flat):+ flat = [x.to("meta") if isinstance(x, torch.Tensor) and not x.is_meta else x for x in flat]+ args, kwargs = tree_unflatten(flat, spec)+ return func(*args, **kwargs)++ return MetaWins+++def meta_reads(enabled: bool):+ """Point safetensors.safe_open and every binding of iter_expert_tensors_parallel at the meta readers."""+ if not enabled:+ yield+ return+ import safetensors++ import freetoken.models.weight as weight++ real_open, real_parallel = safetensors.safe_open, weight.iter_expert_tensors_parallel+ swapped = []+ safetensors.safe_open = _MetaSafeOpen+ for module in list(sys.modules.values()):+ if getattr(module, "__name__", "").startswith("freetoken.") and getattr(module, "iter_expert_tensors_parallel", None) is real_parallel:+ module.iter_expert_tensors_parallel = _meta_iter_expert_tensors_parallel+ swapped.append(module)+ try:+ with _meta_wins()():+ yield+ finally:+ safetensors.safe_open = real_open+ for module in swapped:+ module.iter_expert_tensors_parallel = real_parallel+++def _needs_values(exc: BaseException) -> bool:+ return "meta" in f"{type(exc).__name__}: {exc}".lower()+++# --- load stages (worker) ---+++def _init_worker() -> None:+ os.environ["CUDA_VISIBLE_DEVICES"] = ""+ # the readers' progress bars; errors reach the main process in the record+ sys.stderr = open(os.devnull, "w")+ warnings.filterwarnings("ignore")+ logging.disable(logging.WARNING)+ import torch++ from freetoken.distributed import set_tp_info+ from freetoken.layers import rotary++ set_tp_info(rank=0, size=1)+ rotary.set_rope_device(torch.device("cpu"))+++def _engine_config(path: str, text_only: bool):+ import torch++ from freetoken.distributed import try_get_tp_info+ from freetoken.engine.config import EngineConfig+ from freetoken.mm.config import ENCODER_KINDS, MultimodalConfig++ mm = MultimodalConfig(disabled_encoders=frozenset(ENCODER_KINDS) if text_only else frozenset())+ return EngineConfig(model_path=path, tp_info=try_get_tp_info(), dtype=torch.bfloat16, moe_strategy="offload", mm=mm)+++def _build_model(config):+ import torch++ from freetoken.engine.engine import _decode_target+ from freetoken.layers.quantization import QuantBackend, set_quant_backend+ from freetoken.models import create_model+ from freetoken.utils.torch_utils import torch_dtype++ set_quant_backend(QuantBackend.parse(config.quant_backend))+ object.__setattr__(config.model_config, "moe_strategy", "offload")+ object.__setattr__(config.model_config, "decode_target", _decode_target(config))+ with torch.device("meta"), torch_dtype(torch.bfloat16):+ return create_model(config.model_config)+++class _Recorder(dict):+ """The state dict model.load_state_dict consumes: it records the keys a layer asks for and does not find, and the shapes that differ, instead of stopping at the first.++ The layers' own load_state_dict decide what they take (a tied lm_head pops lm_head.weight itself); what is left at the end is unexpected."""++ def __init__(self, expected: dict):+ super().__init__()+ self.expected, self.missing, self.shapes = expected, [], []++ def pop(self, key, *default):+ import torch++ if key not in self:+ if default:+ return default[0]+ self.missing.append(key)+ return torch.empty_like(self.expected[key], device="meta")+ t = super().pop(key)+ ref = self.expected.get(key)+ if ref is None:+ return t+ if tuple(t.shape) != tuple(ref.shape):+ self.shapes.append(f"{key}: checkpoint {tuple(t.shape)}, model {tuple(ref.shape)}")+ return torch.empty_like(ref, device="meta")+ return t.to(dtype=ref.dtype)+++_LOSSY = {("float8_e4m3fn", "bfloat16"), ("float8_e5m2", "bfloat16"), ("bfloat16", "float8_e4m3fn"), ("float32", "float8_e4m3fn"), ("bfloat16", "uint8")}+++def dense_load(path: str, config, model, *, meta: bool) -> tuple[list[str], list[str], dict]:+ """(problems, warnings, facts) of the engine's dense weight load: what load_weight yields against the model's state dict."""+ import torch++ from freetoken.models import load_weight++ expected = model.state_dict()+ is_moe = bool(getattr(config.model_config, "is_moe", False))+ loaded, dups, casts = _Recorder(expected), [], Counter()+ with meta_reads(meta):+ for key, t in load_weight(path, torch.device("cpu"), include_moe_experts=not is_moe, include_vision=bool(config.active_encoders)):+ ref = expected.get(key)+ # as _materialize_loaded_weight_state_dict: drop an input_scale the layer declares none for, cast to the parameter dtype+ if ref is None and key.endswith(".input_scale"):+ continue+ if key in loaded:+ dups.append(key)+ if ref is not None and (str(t.dtype).removeprefix("torch."), str(ref.dtype).removeprefix("torch.")) in _LOSSY:+ casts[f"{t.dtype} -> {ref.dtype}"] += 1+ # the check keeps no tensor bytes+ dict.__setitem__(loaded, key, t if t.is_meta else torch.empty_like(t, device="meta"))+ try:+ model.load_state_dict(loaded)+ except RuntimeError as e:+ if not str(e).startswith("Unexpected keys"):+ raise+ unexpected, missing, shapes = sorted(loaded), sorted(loaded.missing), loaded.shapes+ problems = []+ if unexpected:+ problems.append(f"{len(unexpected)} unexpected tensors: {group_names(unexpected)}")+ if missing:+ problems.append(f"{len(missing)} model tensors never loaded: {group_names(missing)}")+ if shapes:+ problems.append(f"{len(shapes)} shape mismatches, e.g. {shapes[0]}")+ warns = [f"{n} tensors cast {k} at load (lossy)" for k, n in casts.items()]+ if dups:+ warns.append(f"{len(dups)} tensors are read twice and the engine keeps the last: {group_names(dups)}")+ facts = {"missing_only_scales": bool(missing) and not unexpected and all(m.endswith(_SCALE_ROLES) for m in missing)}+ return problems, warns, facts+++def expert_load(path: str, config, model, *, meta: bool) -> list[str]:+ """Problems of filling every MoE layer's expert banks, as build_expert_banks fills them; nothing is kept."""+ import torch++ from freetoken.engine.engine import offload_layer_methods+ from freetoken.moe.expert_banks import method_groups+ from freetoken.models.weight import experts_scattered+ from freetoken.moe.expert_pieces import iter_expert_pieces++ methods = offload_layer_methods(model)+ if methods is None:+ return []+ mc = config.model_config+ num_layers, num_experts = mc.num_moe_layers, mc.num_experts+ written = torch.zeros(num_layers, num_experts, dtype=torch.int32)+ groups = method_groups(methods)+ device = torch.device("meta" if meta else "cpu")+ # the reader load_expert_banks picks: the parallel one (names from the index) for experts stored as many small tensors+ parallel = experts_scattered(path)++ def fill(method, layers, par: bool) -> str | None:+ keep = set(layers)+ layout = {role: spec for role, spec in method.layout().items() if not spec.resident}+ pieces = iter_expert_pieces(path, mc, method.kind, layers=layers if len(groups) > 1 else None, parallel=par)+ for layer, e0, e1, piece in pieces:+ if not (0 <= layer < num_layers and 0 <= e0 < e1 <= num_experts):+ return f"expert piece out of range: layer {layer}, experts {e0}:{e1} of {num_layers} x {num_experts}"+ if layer not in keep:+ continue+ if written[layer, e0:e1].any():+ return f"expert rows read more than once: layer {layer}, experts {e0}:{e1}"+ written[layer, e0:e1] = 1+ out = {role: torch.empty((e1 - e0, *spec.shape), dtype=spec.dtype, device=device) for role, spec in layout.items()}+ method.pack(piece, out)+ return None++ with meta_reads(meta):+ for method, layers in groups:+ try:+ problem = fill(method, layers, parallel)+ except NotImplementedError:+ # load_expert_banks rebuilds serially when a quant has no parallel reader+ if not parallel:+ raise+ written[layers] = 0+ problem = fill(method, layers, False)+ if problem:+ return [problem]+ missing = (written == 0).nonzero().tolist()+ if missing:+ return [f"expert banks were not filled: {len(missing)} (layer, expert) rows missing (first {missing[:4]})"]+ return []+++# --- transformers cross-check (worker) ---+++def hf_check(path: str, table: dict) -> dict:+ """Whether transformers' own model takes these tensor names, through its checkpoint conversion rules.++ Quantization companions fold into their module's ``.weight``. Returns the names it does not map, the model keys nothing maps to, and the names a WeightConverter repacks."""+ import torch+ import transformers+ from transformers.conversion_mapping import get_model_conversion_mapping+ from transformers.core_model_loading import WeightConverter, WeightRenaming, rename_source_key++ transformers.utils.logging.disable_progress_bar()+ cfg = transformers.AutoConfig.from_pretrained(path)+ arch = (getattr(cfg, "architectures", None) or [None])[0]+ cls = getattr(transformers, arch or "", None)+ if cls is None:+ return {"skipped": f"transformers has no class {arch}"}+ with torch.device("meta"):+ model = cls._from_config(cfg)+ meta_sd = model.state_dict()+ transforms = get_model_conversion_mapping(model)+ renamings = [t for t in transforms if isinstance(t, WeightRenaming)]+ converters = [t for t in transforms if isinstance(t, WeightConverter)]+ ignore_unexpected = [re.compile(p) for p in (getattr(model, "_keys_to_ignore_on_load_unexpected", None) or [])]+ ignore_missing = [re.compile(p) for p in (getattr(model, "_keys_to_ignore_on_load_missing", None) or [])]+ names = set()+ for name in table:+ module, leaf = split_module(name)+ if leaf in QUANT_COMPANIONS:+ continue+ names.add(module + ".weight" if leaf == "weight_packed" else name)+ mapped, converted = {}, set()+ for name in names:+ new, conv = rename_source_key(name, renamings, converters, model.base_model_prefix, meta_sd)+ mapped[name] = new+ if conv is not None:+ converted.add(name)+ unmapped = sorted(n for n, k in mapped.items() if k not in meta_sd and not any(rx.search(k) for rx in ignore_unexpected))+ tied = bool(getattr(cfg.get_text_config(), "tie_word_embeddings", False))+ targets = set(mapped.values())+ missing = sorted(k for k in meta_sd if k not in targets and not any(rx.search(k) for rx in ignore_missing)+ and not (tied and k.endswith("lm_head.weight")))+ renamed = sorted(n for n, k in mapped.items() if k != n and n not in converted and k in meta_sd)+ return {"arch": arch, "unmapped": unmapped, "missing": missing, "converted": sorted(converted), "renamed": renamed}+++# --- lints (worker) ---+++def _skip_entry(dialect: str, modules: list[str], all_modules) -> list[str]:+ """Skip-list entries in the dialect's syntax that cover exactly ``modules`` of ``all_modules``.++ A pattern (layer numbers as a wildcard) stands for its modules only when every module of ``all_modules`` with that pattern is in ``modules``; the others are listed by name."""+ chosen = set(modules)+ by_pattern: dict[str, list[str]] = {}+ for m in all_modules:+ by_pattern.setdefault(pattern_of(m), []).append(m)+ entries = []+ for p in sorted({pattern_of(m) for m in chosen}):+ whole = all(m in chosen for m in by_pattern.get(p, ())) and "N" in p.split(".")+ if whole and dialect == "modelopt":+ entries.append(p.replace(".N", ".*"))+ elif whole and dialect == "compressed-tensors":+ entries.append("re:^" + re.escape(p).replace(r"\.N", r"\.\d+") + "$")+ else:+ entries += sorted(m for m in by_pattern.get(p, [p]) if m in chosen)+ return entries+++def _entry_matches(dialect: str, entry: str, name: str) -> bool:+ from freetoken.layers.quantization.names import ct_set, name_set, substr_set++ if dialect == "compressed-tensors":+ return ct_set((entry,), class_names=False)(name)+ if dialect == "fp8":+ return name_set((entry,))(name) or substr_set((entry,))(name)+ return name_set((entry,))(name)+++def _raw_skip_entries(path: str) -> list[str]:+ from ckpt_edit import SKIP_KEYS, Checkpoint++ entries = []+ for _, q in Checkpoint(path).quant_configs():+ for key in SKIP_KEYS:+ entries += [e for e in q.get(key) or () if isinstance(e, str)]+ return list(dict.fromkeys(entries))+++def lint_quant(path: str, table: dict, spec, hf_config) -> list[dict]:+ """Modules whose stored form contradicts what the quantization_config declares for them, with the skip-list edits that make the two agree."""+ from freetoken.models.register import checkpoint_quant_config++ quant = checkpoint_quant_config(path, hf_config, spec)+ dialect = getattr(quant, "dialect", "none")+ if dialect in ("none", "gguf"):+ return []+ declared_q_stored_f, declared_f_stored_q, exempt_dense = [], [], []+ stored = stored_modules(table)+ for module, (quantized, _) in stored.items():+ # embedding tables (and PLE n-gram tables) are read dense or by their own method whatever the config says+ if any("embed" in seg for seg in module.split(".")) or module.endswith("conv1d"):+ continue+ declared = quant.scheme_for_name(module) is not None+ if quant.unquantized(module):+ if declared and not quantized:+ exempt_dense.append(module)+ elif declared and not quantized:+ declared_q_stored_f.append(module)+ elif quantized and not declared:+ declared_f_stored_q.append(module)+ findings = []+ if exempt_dense:+ findings.append({+ "id": "quant_unlisted_dense_router", "level": "config", "category": "loader",+ "message": f"{len(exempt_dense)} modules FreeToken serves in bf16 anyway are stored bf16 and not listed as skipped: {group_names(exempt_dense)}; listing them works around a reader that wants the stored form to match the config",+ "fix": {"op": "skip_add", "entries": _skip_entry(dialect, exempt_dense, stored), "auto": True},+ })+ targets = _lint_ct_targets(path, table, dialect)+ if declared_q_stored_f:+ findings.append({+ "id": "quant_declared_but_dense", "level": "config", "category": "checkpoint",+ "message": f"{len(declared_q_stored_f)} modules are stored unquantized but the quantization_config does not list them as skipped: {group_names(declared_q_stored_f)}",+ "fix": {"op": "skip_add", "entries": _skip_entry(dialect, declared_q_stored_f, stored), "auto": True},+ })+ # targets that name no module explain the quantized modules nothing covers; recheck after fixing them+ if declared_f_stored_q and not targets:+ entries = _raw_skip_entries(path)+ remove, add = [], []+ for e in entries:+ hits = [m for m in declared_f_stored_q if _entry_matches(dialect, e, m)]+ if not hits:+ continue+ remove.append(e)+ # an entry naming a container also covers its dense children: list those alone instead+ dense = [m for m, (q, _) in stored.items() if not q and _entry_matches(dialect, e, m)]+ add += _skip_entry(dialect, dense, stored) if dense else []+ msg = f"{len(declared_f_stored_q)} modules are stored quantized but the quantization_config skips them or no target covers them: {group_names(declared_f_stored_q)}"+ finding = {"id": "quant_skipped_but_quantized", "level": "config", "category": "checkpoint", "message": msg}+ if remove:+ finding["message"] += f"; skip entries that cover them: {remove[:6]}"+ finding["fixes"] = [{"op": "skip_remove", "entries": remove, "auto": True}] + ([{"op": "skip_add", "entries": add, "auto": True}] if add else [])+ findings.append(finding)+ return findings + targets+++def _lint_ct_targets(path: str, table: dict, dialect: str) -> list[dict]:+ if dialect != "compressed-tensors":+ return []+ from ckpt_edit import Checkpoint++ modules = list(stored_modules(table))+ dead: dict[tuple[str, str], list[str]] = {}+ for _, q in Checkpoint(path).quant_configs():+ for group in (q.get("config_groups") or {}).values():+ for target in group.get("targets") or ():+ if target == "Linear" or any(_entry_matches(dialect, target, m) for m in modules):+ continue+ for pat, repl in TARGET_SWAPS:+ swapped = re.sub(pat, repl, target)+ if swapped != target and any(_entry_matches(dialect, swapped, m) for m in modules):+ dead.setdefault((pat, repl), []).append(target)+ break+ return [{+ "id": "ct_target_vllm_names", "level": "config", "category": "checkpoint",+ "message": f"{len(ts)} compressed-tensors targets name modules as vLLM does (language_model.model.) and match no tensor, e.g. {ts[0]!r}",+ "fix": {"op": "quant_sub", "pattern": pat, "repl": repl, "auto": True},+ } for (pat, repl), ts in dead.items()]+++def rename_rules(findings: list[dict]) -> list[tuple[str, str]]:+ return [tuple(r) for f in findings if f["level"] == "rename" for r in f["fix"]["rules"]]+++def renamed_table(table: dict, rules) -> dict:+ """The tensor table as it reads after the rename rules."""+ out = {}+ for name, meta in table.items():+ for pat, repl in rules:+ name = re.sub(pat, repl, name)+ out[name] = meta+ return out+++def lint_kv_scales(table: dict, spec) -> list[dict]:+ hits = [n for n in table if re.search(KV_SCALE_RULE[0], n)]+ if not hits or spec.module not in KV_SCALE_DROPPERS:+ return []+ return [{+ "id": "kv_output_scale", "level": "rename", "category": "loader",+ "message": f"{len(hits)} fp8 KV-cache scales stored as <k|v>_proj.output_scale ({group_names(hits, 2)}); vLLM reads them as k_scale / v_scale, FreeToken drops k_scale / v_scale but not output_scale",+ "fix": {"op": "rename", "rules": [list(KV_SCALE_RULE)], "auto": True},+ }]+++def lint_prefixes(table: dict, spec) -> list[dict]:+ roots = {ckpt for _, ckpt in spec.checkpoint_roots}+ names = list(table)+ findings = []+ for pat, repl, why, root in PREFIX_RULES:+ if root not in roots:+ continue+ rx = re.compile(pat)+ hits = [n for n in names if rx.search(n)]+ if not hits:+ continue+ renamed = {rx.sub(repl, n) for n in hits}+ if renamed & (set(names) - set(hits)):+ continue+ findings.append({+ "id": "prefix", "level": "rename", "category": "checkpoint",+ "message": f"{why}: {len(hits)} tensors {rx.search(hits[0]).group(0)} -> {repl}",+ "fix": {"op": "rename", "rules": [[pat, repl]], "quant_too": True, "auto": True},+ })+ return findings+++def lint_experts(table: dict, num_experts: int | None) -> list[dict]:+ """Per-expert tensors (experts.E.<proj>) where some layer lacks some expert's projection: a checkpoint that lost tensors, not a packing form."""+ if not num_experts:+ return []+ seen: dict[tuple[str, str, str], set[int]] = {}+ for name in table:+ m = _PER_EXPERT.match(name)+ if m:+ seen.setdefault((m["pre"] + m["layer"], m["mid"], m["proj"]), set()).add(int(m["e"]))+ gaps = [(f"{pre}{mid}N.{proj}", sorted(set(range(num_experts)) - es)) for (pre, mid, proj), es in seen.items() if len(es) < num_experts]+ if not gaps:+ return []+ shown = "; ".join(f"{where} lacks experts {missing[:6]}" for where, missing in gaps[:3])+ return [{"id": "expert_tensors_incomplete", "level": "none", "category": "checkpoint",+ "message": f"{len(gaps)} (layer, projection) groups of per-expert tensors are incomplete: {shown}; the checkpoint is missing tensors (download it again, or ask its uploader)"}]+++def lint_vision(path: str, table: dict, spec, text_only: bool) -> list[dict]:+ from freetoken.models.config import VISION_KEY_PREFIXES++ if text_only or not spec.encoders:+ return []+ # checkpoint names put the tower under a root (model.visual., model.vision_tower.)+ segments = tuple("." + p for p in VISION_KEY_PREFIXES)+ if not any(("." + n).count(s) for n in table for s in segments):+ return [{"id": "no_vision_tower", "level": "runtime", "category": "checkpoint",+ "message": "the architecture has a vision tower but the checkpoint stores none: a text-only checkpoint under a multimodal architecture; serve it with --text-model-only (and diagnose it with --text-model-only)"}]+ if spec.mm_processor and not any(os.path.isfile(os.path.join(path, f)) for f in ("preprocessor_config.json", "processor_config.json")):+ return [{"id": "no_processor_config", "level": "file", "category": "checkpoint",+ "message": "the checkpoint has a vision tower but no preprocessor_config.json, so image requests fail with a 400; copy that file from the base model the checkpoint was tuned from"}]+ return []+++def lint_misc(table: dict, hf_config) -> list[dict]:+ from freetoken.layers.quantization.configs import quantization_config_of++ out = []+ dtypes = Counter(m["dtype"] for m in table.values())+ if quantization_config_of(hf_config) is None and (dtypes["F8_E4M3"] or dtypes["F8_E5M2"]):+ out.append({"id": "fp8_without_config", "level": "none", "category": "checkpoint",+ "message": f"{dtypes['F8_E4M3'] + dtypes['F8_E5M2']} fp8 tensors but no quantization_config says how to read them (scales, block size)"})+ if any(n.endswith((".scales", ".biases")) for n in table) and any(m["dtype"] == "U32" for m in table.values()):+ out.append({"id": "mlx", "level": "none", "category": "quant_method", "message": "MLX-quantized weights (.scales / .biases, U32 packing); FreeToken does not read MLX"})+ return out+++def expert_layout(table: dict) -> str | None:+ per_expert = any(re.search(r"\.experts\.\d+\.(gate|up|down|w[123])", n) for n in table)+ stacked = any(re.search(r"\.experts\.(gate_up_proj|down_proj|w13_weight|w2_weight)$", n) for n in table)+ if per_expert and stacked:+ return "mixed: some layers per expert, some stacked"+ return "per expert (experts.N.<proj>)" if per_expert else "stacked (experts.gate_up_proj / down_proj)" if stacked else None+++# --- classification ---++_QUANT_METHOD = re.compile(r"quantization method .* is not supported|quant_algo .* is not supported|compressed-tensors .*scheme .* is not supported|MIXED_PRECISION without|fp8 checkpoint with weight_block_size|fp8 block size .* is not supported|target class .* is not supported")+_QUANT_MODEL = re.compile(r"no quant method for|provides no (expert reader|nvfp4_expert_spec|iter_weights_parallel)|needs in/out sizes divisible|is not supported for|cannot use the .* kernel")+++def classify(rec: dict) -> str:+ """The category of a failed record (README.md, "Safetensors: six kinds of failure"), first match wins."""+ first = (rec["problems"] or [""])[0]+ findings = rec["findings"]+ hf = rec.get("transformers") or {}+ if rec["stage"] == "files":+ return "checkpoint"+ if rec["stage"] == "config" and "architecture" in first.lower() and "not supported" in first:+ return "arch"+ if any(f["category"] == "quant_method" for f in findings) or _QUANT_METHOD.search(first):+ return "quant_method"+ # a broken prefix misnames every module under it, so whatever fails next is its symptom+ if any(f["id"] == "prefix" and f["category"] == "checkpoint" for f in findings):+ return "checkpoint"+ if _QUANT_MODEL.search(first) or rec.get("missing_only_scales"):+ return "quant_model"+ if hf.get("unmapped") or hf.get("missing"):+ return "checkpoint"+ if any(f["category"] == "checkpoint" for f in findings):+ return "checkpoint"+ if rec["stage"] == "experts" and hf.get("converted"):+ return "packing"+ return "loader"+++def _fix_list(findings: list[dict]) -> list[dict]:+ fixes = []+ for f in findings:+ for fix in f.get("fixes") or ([f["fix"]] if "fix" in f else []):+ fixes.append({"level": f["level"], "finding": f["id"], **fix})+ return fixes+++def run_lints(path: str, table: dict, spec, hf_config, text_only: bool, warns: list[str], num_experts: int | None = None) -> list[dict]:+ """Every lint; the quantization lints see the names as they read after the rename fixes, which a rename applies first."""+ def safe(fn, *args):+ try:+ return fn(*args)+ except Exception as e:+ warns.append(f"lint {fn.__name__} failed: {type(e).__name__}: {str(e)[:200]}")+ return []++ if spec is None:+ return safe(lint_misc, table, hf_config) if hf_config is not None else []+ findings = safe(lint_prefixes, table, spec) + safe(lint_kv_scales, table, spec)+ after = renamed_table(table, rename_rules(findings))+ findings += safe(lint_experts, after, num_experts) + safe(lint_vision, path, after, spec, text_only) + safe(lint_misc, after, hf_config)+ return findings + safe(lint_quant, path, after, spec, hf_config)+++class _Stop(Exception):+ """A stage found problems of its own; the later stages would only repeat them."""+++def check(path: str, text_only: bool) -> dict:+ rec = {"path": path, "stage": "config", "status": "fail", "problems": [], "warnings": [], "findings": []}+ stub = _stub_record(path)+ if stub:+ rec["repo"], rec["revision"] = stub["repo"], stub["revision"]+ t0 = time.time()+ hf_config = spec = table = None+ try:+ from freetoken.models.register import get_model_spec+ from freetoken.utils import cached_load_hf_config++ rec["stage"] = "files"+ table = read_tensor_table(path)+ file_problems = check_files(path)+ if file_problems:+ rec["problems"] += file_problems+ raise _Stop+ rec["stage"] = "config"+ hf_config = cached_load_hf_config(path)+ rec["arch"] = (getattr(hf_config, "architectures", None) or [None])[0]+ spec = get_model_spec(rec["arch"])+ rec["stage"] = "quant"+ config = _engine_config(path, text_only)+ mc = config.model_config+ rec["dialect"] = getattr(getattr(mc, "quant", None), "dialect", None)+ rec["moe"] = {"layers": mc.num_moe_layers, "experts": mc.num_experts, "layout": expert_layout(table)} if getattr(mc, "is_moe", False) else None+ rec["encoders"] = [e.kind for e in config.active_encoders]+ rec["stage"] = "build"+ model = _build_model(config)+ for stage, fn in (("dense", dense_load), ("experts", expert_load)):+ rec["stage"] = stage+ try:+ out = fn(path, config, model, meta=True)+ except Exception as e:+ if not _needs_values(e):+ raise+ rec.setdefault("real_reads", []).append(stage)+ out = fn(path, config, model, meta=False)+ problems, warns, facts = out if stage == "dense" else (out, [], {})+ rec["warnings"] += warns+ rec.update(facts)+ if problems:+ rec["problems"] += problems+ break+ else:+ rec["stage"], rec["status"] = "done", "pass"+ except _Stop:+ pass+ except Exception as e:+ rec["problems"].append(f"{type(e).__name__}: {str(e)[:600]}")+ rec["traceback"] = traceback.format_exc()[-2500:]+ if table is not None:+ rec["findings"] = run_lints(path, table, spec, hf_config, text_only, rec["warnings"], (rec.get("moe") or {}).get("experts"))+ if rec["status"] == "fail" and table is not None and rec["stage"] in ("build", "dense", "experts"):+ try:+ rec["transformers"] = hf_check(path, table)+ except Exception as e:+ rec["transformers"] = {"skipped": f"{type(e).__name__}: {str(e)[:200]}"}+ hf = rec["transformers"]+ if "skipped" not in hf and not hf["unmapped"] and not hf["missing"]:+ # transformers reads these names, so a rename is a workaround for the loader rather than a repair+ for f in rec["findings"]:+ if f["id"] == "prefix":+ f["category"] = "loader"+ if rec["status"] == "fail":+ rec["category"] = classify(rec)+ # nothing to repair in the files of a checkpoint whose format or architecture is not served+ fixes = [] if rec.get("category") in ("arch", "quant_method") else _fix_list(rec["findings"])+ # a remote repo is only diagnosed: fixes are made on a local download, diagnosed again+ rec["fixes"] = [] if stub else fixes+ rec["fixable_locally"] = bool(stub and fixes)+ rec["seconds"] = round(time.time() - t0, 1)+ return rec+++# --- main process ---+++def resolve(target: str, stub_dir: str) -> tuple[str | None, dict | None]:+ """(local path, None), or (None, error record) when a repo cannot be fetched."""+ if os.path.isdir(target):+ return os.path.abspath(target), None+ from fetch_headers import fetch_repo++ try:+ return fetch_repo(target, stub_dir)["stub"], None+ except Exception as e:+ return None, {"path": target, "status": "skip", "format": "unknown", "hint": f"not a local directory, and fetching it from HF failed: {type(e).__name__}: {str(e)[:300]}"}+++def _short(rec: dict) -> str:+ return rec.get("repo") or rec["path"]+++def print_record(rec: dict, report_path: str | None) -> None:+ print(f"\n== {_short(rec)}" + (" (header-only stub)" if rec.get("repo") else ""))+ if rec["status"] == "skip":+ print(f" {rec['hint']}")+ return+ bits = [rec.get("arch") or "?", f"quant {rec.get('dialect') or 'none'}"]+ if rec.get("moe"):+ bits.append(f"MoE {rec['moe']['layers']} layers x {rec['moe']['experts']} experts, {rec['moe']['layout'] or 'experts not found by name'}")+ print(" " + ", ".join(bits))+ if rec["status"] == "pass":+ print(" PASS: config, build, dense load" + (", expert banks" if rec.get("moe") else ""))+ else:+ cat = rec["category"]+ print(f" FAIL at {rec['stage']}: [{cat}] {CATEGORIES[cat]}")+ for p in rec["problems"]:+ print(f" - {p}")+ hf = rec.get("transformers") or {}+ if hf.get("skipped"):+ print(f" transformers check skipped: {hf['skipped']}")+ elif hf:+ if hf["unmapped"] or hf["missing"]:+ parts = []+ if hf["unmapped"]:+ parts.append(f"{len(hf['unmapped'])} checkpoint names it cannot map ({group_names(hf['unmapped'], 3)})")+ if hf["missing"]:+ parts.append(f"{len(hf['missing'])} model keys nothing fills ({group_names(hf['missing'], 3)})")+ print(f" transformers {hf['arch']} would not load it either: " + "; ".join(parts))+ else:+ extra = []+ if hf["renamed"]:+ extra.append(f"renames {len(hf['renamed'])} ({group_names(hf['renamed'], 2)})")+ if hf["converted"]:+ extra.append(f"repacks {len(hf['converted'])} ({group_names(hf['converted'], 2)})")+ print(" transformers loads these names" + (": " + "; ".join(extra) if extra else ""))+ for w in rec["warnings"]:+ print(f" warning: {w}")+ for f in rec["findings"]:+ print(f" {'note' if rec['status'] == 'pass' else 'finding'} [{f['level']}, {f['category']}]: {f['message']}")+ if rec.get("fixable_locally") and rec["status"] == "fail":+ print(" fixable: download the checkpoint and run diagnose.py on the local directory to get the fix commands (a remote repo is only diagnosed)")+ fixes = rec.get("fixes") or []+ if not fixes or rec["status"] != "fail":+ return+ target, report_arg = rec["path"], report_path or "<report.jsonl>"+ # one level per round: a rename changes the names the config findings are about, so diagnose again after it+ level = "rename" if any(f["level"] == "rename" and f.get("auto") for f in fixes) else "config"+ if any(f["level"] == level and f.get("auto") for f in fixes):+ if not report_path:+ print(" fixes: rerun with --out <report.jsonl> to get the fix command")+ return+ print(f" next step ({level} fix; it prints a plan and asks before writing):")+ print(f" python scripts/checkpoint_diagnose/{FIX_SCRIPTS[level]} {target} --out <new dir> --from-report {report_arg}")+ print(" then diagnose <new dir> again")+++def summary(records: list[dict]) -> None:+ status = Counter(r["status"] for r in records)+ print(f"\n{status['pass']} pass, {status['fail']} fail, {status['skip']} not checked")+ cats = Counter(r["category"] for r in records if r["status"] == "fail")+ for cat in CATEGORIES:+ if cats[cat]:+ print(f" {cats[cat]:4d} [{cat}] {CATEGORIES[cat]}")+ for r in records:+ if r["status"] == "fail" and r["category"] == cat:+ fixable = [lv for lv in FIX_SCRIPTS if any(f["level"] == lv and f.get("auto") for f in r.get("fixes", []))]+ note = f" (fixes: {', '.join(fixable)})" if fixable else " (fixable on a local download)" if r.get("fixable_locally") else ""+ print(f" {_short(r)}{note}")+++def main() -> int:+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)+ ap.add_argument("targets", nargs="+", help="checkpoint directories or HF repo ids")+ ap.add_argument("--out", metavar="JSONL", help="append the records here; the fix scripts read them with --from-report")+ ap.add_argument("--text-model-only", action="store_true", help="check as `ft serve --text-model-only` loads: no vision tower")+ args = ap.parse_args()++ records = []+ out = open(args.out, "a") if args.out else None+ ctx = mp.get_context("spawn")+ pool = None+ for target in args.targets:+ path, rec = resolve(target, STUB_DIR)+ rec = rec or route(path)+ if rec is None:+ # one worker process, reused; a crash takes only the checkpoint it was checking+ pool = pool or cf.ProcessPoolExecutor(1, mp_context=ctx, initializer=_init_worker)+ try:+ rec = pool.submit(check, path, args.text_model_only).result()+ except cf.process.BrokenProcessPool:+ pool = None+ rec = {"path": path, "status": "fail", "stage": "?", "category": "loader", "problems": ["the check process died (out of host memory, or a crash)"],+ "warnings": [], "findings": [], "fixes": []}+ records.append(rec)+ print_record(rec, args.out)+ if out:+ out.write(json.dumps(rec) + "\n")+ out.flush()+ if pool:+ pool.shutdown()+ if out:+ out.close()+ if len(records) > 1:+ summary(records)+ return 0 if all(r["status"] != "fail" for r in records) else 1+++if __name__ == "__main__":+ sys.exit(main())diff --git a/scripts/checkpoint_diagnose/fetch_headers.py b/scripts/checkpoint_diagnose/fetch_headers.pynew file mode 100644index 00000000..da2ae716--- /dev/null+++ b/scripts/checkpoint_diagnose/fetch_headers.py@@ -0,0 +1,137 @@+"""Write header-only stubs of HF safetensors checkpoints: the config files and every shard's header, in sparse files of the real size.++ python scripts/checkpoint_diagnose/fetch_headers.py <out_dir> <repo>[@<revision>] ...++The tensor data stays holes that read as zeros, so a stub takes a few MB of disk and ``diagnose.py`` reads it like the real checkpoint.+Stubs land at ``<out_dir>/<repo>``; a rerun keeps the files that already have the right size.+Python files are never fetched. Gated repos read ``HF_TOKEN``. Prints one JSON line per repo.+"""++from __future__ import annotations++import argparse+import concurrent.futures as cf+import json+import os+import struct+import sys+import threading+import time++import requests+from huggingface_hub import HfApi, hf_hub_url+from huggingface_hub.utils import build_hf_headers++STUB_MARKER = ".stub.json"+CONFIG_EXT = (".json", ".jinja")+# large tokenizer files: the load check never reads them+SKIP = {"tokenizer.json", "vocab.json", "merges.txt"}+_FIRST_READ = 1 << 20++_local = threading.local()+++def _session() -> requests.Session:+ if not hasattr(_local, "session"):+ _local.session = requests.Session()+ _local.session.headers.update(build_hf_headers())+ return _local.session+++def _get(url: str, start: int | None = None, end: int | None = None) -> bytes:+ """The file, or bytes [start, end] of it; retries rate limits and server errors."""+ headers = {} if start is None else {"Range": f"bytes={start}-{end}"}+ for attempt in range(6):+ r = _session().get(url, headers=headers, timeout=300)+ if r.status_code == 429 or r.status_code >= 500:+ time.sleep(5 * (attempt + 1))+ continue+ r.raise_for_status()+ if start is not None and r.status_code != 206:+ raise ValueError(f"{url}: asked for a byte range, got status {r.status_code}")+ return r.content+ r.raise_for_status()+ raise RuntimeError(f"{url}: status {r.status_code}")+++def _fetch_shard(url: str, dst: str, size: int) -> None:+ if os.path.isfile(dst) and os.path.getsize(dst) == size:+ return+ head = _get(url, 0, min(size, _FIRST_READ) - 1)+ (n,) = struct.unpack("<Q", head[:8])+ if 8 + n > len(head):+ head += _get(url, len(head), 8 + n - 1)+ head = head[: 8 + n]+ json.loads(head[8:])+ tmp = dst + ".part"+ with open(tmp, "wb") as f:+ f.write(head)+ f.truncate(size)+ os.replace(tmp, dst)+++def _fetch_file(url: str, dst: str, size: int | None) -> None:+ if os.path.isfile(dst) and size is not None and os.path.getsize(dst) == size:+ return+ data = _get(url)+ with open(dst + ".part", "wb") as f:+ f.write(data)+ os.replace(dst + ".part", dst)+++def fetch_repo(repo: str, out_dir: str, *, workers: int = 8) -> dict:+ """Write the stub of ``repo`` (``name`` or ``name@revision``) under ``out_dir``; returns its record, with ``stub`` the directory."""+ name, _, revision = repo.partition("@")+ info = HfApi().model_info(name, revision=revision or None, files_metadata=True)+ files = [(s.rfilename, s.size) for s in info.siblings or []]+ root = [(f, size) for f, size in files if "/" not in f]+ shards = [(f, size) for f, size in root if f.endswith(".safetensors")]+ configs = [(f, size) for f, size in root if f.endswith(CONFIG_EXT) and f not in SKIP]+ dst = os.path.join(out_dir, name)+ rec = {"repo": name, "revision": info.sha, "stub": dst, "shards": len(shards),+ "gguf": sum(f.endswith(".gguf") for f, _ in files),+ "other_weights": sorted({os.path.splitext(f)[1] for f, _ in files if f.endswith((".bin", ".pt", ".pth", ".npz"))}),+ "files": [f for f, _ in configs + shards]}+ marker = os.path.join(dst, STUB_MARKER)+ if os.path.isfile(marker):+ with open(marker) as f:+ old = json.load(f)+ if old.get("revision") != info.sha:+ # another revision: none of its files can be kept+ for f in old.get("files", []):+ if os.path.isfile(os.path.join(dst, f)):+ os.remove(os.path.join(dst, f))+ os.remove(marker)+ os.makedirs(dst, exist_ok=True)+ jobs = [(_fetch_file, f, size) for f, size in configs] + [(_fetch_shard, f, size) for f, size in shards]+ with cf.ThreadPoolExecutor(workers) as ex:+ futures = [ex.submit(fn, hf_hub_url(name, f, revision=info.sha), os.path.join(dst, f), size) for fn, f, size in jobs]+ for fut in futures:+ fut.result()+ with open(os.path.join(dst, STUB_MARKER), "w") as f:+ json.dump(rec, f, indent=1)+ return rec+++def is_stub(path: str) -> bool:+ return os.path.isfile(os.path.join(path, STUB_MARKER))+++def main() -> int:+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)+ ap.add_argument("out_dir")+ ap.add_argument("repos", nargs="+", help="HF repo ids, optionally with @revision")+ args = ap.parse_args()+ failed = 0+ for repo in args.repos:+ try:+ rec = {**fetch_repo(repo, args.out_dir), "ok": True}+ except Exception as e:+ rec = {"repo": repo, "ok": False, "error": f"{type(e).__name__}: {e}"[:300]}+ failed += 1+ print(json.dumps(rec), flush=True)+ return 1 if failed else 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/scripts/checkpoint_diagnose/fix_config.py b/scripts/checkpoint_diagnose/fix_config.pynew file mode 100644index 00000000..93bda8af--- /dev/null+++ b/scripts/checkpoint_diagnose/fix_config.py@@ -0,0 +1,102 @@+"""Level 1 fix: edit only config.json / hf_quant_config.json of a local HF safetensors checkpoint (a remote repo's stub is refused).++ python scripts/checkpoint_diagnose/fix_config.py <ckpt> --out <new dir> --from-report report.jsonl+ python scripts/checkpoint_diagnose/fix_config.py <ckpt> --out <new dir> --set text_config.tie_word_embeddings false+ python scripts/checkpoint_diagnose/fix_config.py <ckpt> --in-place --skip-add 'model.layers.*.mlp.gate' --plan plan.json+ python scripts/checkpoint_diagnose/fix_config.py <ckpt> --restore++Edits, applied in this order:+ --from-report JSONL the automatic config fixes diagnose.py --out found for this checkpoint+ --set PATH VALUE a config.json key; PATH is dotted, VALUE is JSON (false, 4, "text")+ --skip-add ENTRY add a module to the quantization skip lists (ignore / exclude_modules / modules_to_not_convert)+ --skip-remove ENTRY remove an entry from those lists++Write a skip entry in the checkpoint dialect's syntax: a glob for ModelOpt (model.layers.*.mlp.gate), a module name or ``re:<regex>`` for compressed-tensors, a module name for fp8.+Every run prints the plan first and asks [y/N]; with ``--plan FILE`` the first run only saves the plan and the second identical run applies it.+"""++from __future__ import annotations++import argparse+import json+import os+import sys++sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))+from ckpt_edit import Checkpoint, check_out_dir, confirm, plan_lines_for_save, refuse_stub, restore # noqa: E402+++def report_fixes(report: str, ckpt: str, level: str) -> list[dict]:+ """The automatic ``level`` fixes in the latest diagnose.py record of ``ckpt`` (matched by path or repo id)."""+ with open(report) as f:+ recs = [json.loads(line) for line in f if line.strip()]+ mine = [r for r in recs if os.path.realpath(r.get("path", "")) == os.path.realpath(ckpt) or r.get("repo") == ckpt]+ if not mine:+ raise SystemExit(f"{report} has no record of {ckpt}: run diagnose.py {ckpt} --out {report} first")+ if mine[-1].get("status") != "fail":+ raise SystemExit(f"{ckpt} passed in {report}: its notes are not fixes to apply")+ return [f for f in mine[-1].get("fixes", []) if f["level"] == level and f.get("auto")]+++def apply_fix(ck: Checkpoint, fix: dict) -> None:+ op = fix["op"]+ if op == "set":+ ck.set_config(fix["path"], fix["value"])+ elif op == "skip_add":+ ck.quant_skip_add(fix["entries"])+ elif op == "skip_remove":+ ck.quant_skip_remove(fix["entries"])+ elif op == "quant_sub":+ ck.quant_sub(fix["pattern"], fix["repl"])+ else:+ raise ValueError(f"unknown config op {op!r}")+++def main() -> int:+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)+ ap.add_argument("ckpt")+ where = ap.add_mutually_exclusive_group(required=True)+ where.add_argument("--out", help="write the fixed checkpoint into this new directory")+ where.add_argument("--in-place", action="store_true", help="edit the checkpoint itself, keeping a backup")+ where.add_argument("--restore", action="store_true", help="undo the latest --in-place edit")+ ap.add_argument("--from-report", metavar="JSONL")+ ap.add_argument("--set", nargs=2, action="append", default=[], metavar=("PATH", "VALUE"))+ ap.add_argument("--skip-add", action="append", default=[], metavar="ENTRY")+ ap.add_argument("--skip-remove", action="append", default=[], metavar="ENTRY")+ ap.add_argument("--plan", metavar="FILE", help="no prompt: the first run saves the plan here, the second identical run applies it")+ args = ap.parse_args()++ refuse_stub(args.ckpt)+ if args.restore:+ for name in restore(args.ckpt):+ print(f"restored {name}")+ return 0+ fixes = report_fixes(args.from_report, args.ckpt, "config") if args.from_report else []+ if args.from_report and not fixes and not (args.set or args.skip_add or args.skip_remove):+ raise SystemExit(f"the record of {args.ckpt} in {args.from_report} has no automatic config fix")+ fixes += [{"op": "set", "path": p, "value": json.loads(v)} for p, v in args.set]+ fixes += [{"op": "skip_add", "entries": args.skip_add}] if args.skip_add else []+ fixes += [{"op": "skip_remove", "entries": args.skip_remove}] if args.skip_remove else []+ if not fixes:+ ap.error("give --from-report, --set, --skip-add or --skip-remove")+ ck = Checkpoint(args.ckpt)+ check_out_dir(ck, args.out)+ for fix in fixes:+ apply_fix(ck, fix)+ if not ck.log:+ print("nothing to change: the config already reads this way")+ return 0+ lines = [f"plan: edit the config of {ck.path}"] + [f" {line}" for line in ck.log] + plan_lines_for_save(ck, args.out)+ plan = {"tool": "fix_config", "ckpt": ck.path, "fingerprint": ck.fingerprint, "out": args.out and os.path.abspath(args.out), "log": ck.log}+ if not confirm(plan, lines, args.plan):+ return 0+ for name in ck.save(out=None if args.in_place else args.out):+ print(f"wrote {name}")+ if args.plan:+ os.remove(args.plan)+ print("done: run diagnose.py on the result to check it")+ return 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/scripts/checkpoint_diagnose/rename_tensors.py b/scripts/checkpoint_diagnose/rename_tensors.pynew file mode 100644index 00000000..74cc040d--- /dev/null+++ b/scripts/checkpoint_diagnose/rename_tensors.py@@ -0,0 +1,123 @@+"""Level 2 fix: rename tensors of a local HF safetensors checkpoint (a remote repo's stub is refused). Only the shard headers, the index and the quantization skip lists change; tensor data is kept byte for byte.++ python scripts/checkpoint_diagnose/rename_tensors.py <ckpt> --out <new dir> --rule PATTERN REPL [--rule ...]+ python scripts/checkpoint_diagnose/rename_tensors.py <ckpt> --out <new dir> --from-report report.jsonl+ python scripts/checkpoint_diagnose/rename_tensors.py <ckpt> --in-place ... --plan plan.json+ python scripts/checkpoint_diagnose/rename_tensors.py <ckpt> --restore++A rule is ``re.sub(PATTERN, REPL, name)``; rules apply in order. They are also applied to the quantization skip-list entries and compressed-tensors targets that are module names or globs; ``re:`` entries are not rewritten.+Every run prints the plan first: per rule how many tensors it renames with examples, the skip entries that still name the old modules, and what each shard write costs. Then it asks [y/N].+With ``--plan FILE`` it never asks: the first run saves the plan to FILE and changes nothing, the second identical run applies it.+Refused: a rule that renames nothing, a rename that gives two tensors one name, an ``--out`` that is not empty.+"""++from __future__ import annotations++import argparse+import os+import re+import sys+from collections import Counter++sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))+from ckpt_edit import Checkpoint, check_out_dir, confirm, plan_lines_for_save, refuse_stub, restore # noqa: E402+from fix_config import report_fixes # noqa: E402++_DIGITS = re.compile(r"\.\d+(?=\.|$)")+++def _pattern(name: str) -> str:+ return _DIGITS.sub(".N", name)+++def rule_effects(names: list[str], rules: list[tuple[str, str]]) -> list[dict]:+ """Per rule, applied in order: the names it changes as pattern pairs, and how many of its matches are not at the start of the name."""+ effects, current = [], list(names)+ for pat, repl in rules:+ rx = re.compile(pat)+ pairs, mid = Counter(), 0+ for i, name in enumerate(current):+ new = rx.sub(repl, name)+ if new != name:+ pairs[(_pattern(name), _pattern(new))] += 1+ m = rx.search(name)+ mid += m is not None and m.start() > 0+ current[i] = new+ anchored = pat.startswith("^") or pat.endswith("$")+ effects.append({"rule": [pat, repl], "renamed": sum(pairs.values()), "pairs": pairs, "mid": 0 if anchored else mid})+ return effects+++def main() -> int:+ ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)+ ap.add_argument("ckpt")+ where = ap.add_mutually_exclusive_group(required=True)+ where.add_argument("--out", help="write the renamed checkpoint into this new directory")+ where.add_argument("--in-place", action="store_true", help="edit the checkpoint itself, keeping a backup")+ where.add_argument("--restore", action="store_true", help="undo the latest --in-place edit")+ ap.add_argument("--rule", nargs=2, action="append", default=[], metavar=("PATTERN", "REPL"))+ ap.add_argument("--from-report", metavar="JSONL", help="the automatic rename fixes diagnose.py --out found for this checkpoint")+ ap.add_argument("--plan", metavar="FILE", help="no prompt: the first run saves the plan here, the second identical run applies it")+ args = ap.parse_args()++ refuse_stub(args.ckpt)+ if args.restore:+ for name in restore(args.ckpt):+ print(f"restored {name}")+ return 0+ rules = []+ if args.from_report:+ for fix in report_fixes(args.from_report, args.ckpt, "rename"):+ rules += [tuple(r) for r in fix["rules"]]+ if not rules and not args.rule:+ raise SystemExit(f"the record of {args.ckpt} in {args.from_report} has no automatic rename fix")+ rules += [tuple(r) for r in args.rule]+ if not rules:+ ap.error("give --rule PATTERN REPL or --from-report")+ ck = Checkpoint(args.ckpt)+ check_out_dir(ck, args.out)++ names = list(ck.tensors())+ effects = rule_effects(names, rules)+ lines = [f"plan: rename tensors of {ck.path} ({len(names)} tensors)"]+ for i, e in enumerate(effects, 1):+ pat, repl = e["rule"]+ lines.append(f" rule {i}: s/{pat}/{repl}/ renames {e['renamed']} tensors")+ for (old, new), n in e["pairs"].most_common(6):+ lines.append(f" {old} -> {new}" + (f" (x{n})" if n > 1 else ""))+ if len(e["pairs"]) > 6:+ lines.append(f" ... {len(e['pairs']) - 6} more name patterns")+ if e["mid"]:+ lines.append(f" warning: {e['mid']} of these matches are inside the name, not at its start; anchor the pattern with ^ or $ if that is not meant")+ dead = [i for i, e in enumerate(effects, 1) if not e["renamed"]]+ if dead:+ print("\n".join(lines))+ raise SystemExit(f"refused: rule {', '.join(map(str, dead))} renames no tensor (check the pattern against the names above)")++ try:+ changes = ck.rename_regex(rules)+ except ValueError as e:+ raise SystemExit(f"refused: {e}")+ for pat, repl in rules:+ ck.quant_rename(pat, repl)+ lines += [f" {line}" for line in ck.log]+ lines.append(f" {len(changes)} tensors renamed, {len(names) - len(changes)} keep their names")+ old_modules = {n.rpartition(".")[0] for n in changes}+ new_modules = {n.rpartition(".")[0] for n in ck.tensors()}+ for entry in ck.stale_quant_entries(old_modules, new_modules):+ lines.append(f" warning: {entry} still names the old modules and none of the new ones; fix it with fix_config.py after this rename")+ lines += plan_lines_for_save(ck, args.out)+ plan = {"tool": "rename_tensors", "ckpt": ck.path, "fingerprint": ck.fingerprint, "out": args.out and os.path.abspath(args.out),+ "rules": [list(r) for r in rules], "renamed": len(changes), "log": ck.log}+ if not confirm(plan, lines, args.plan):+ return 0+ for name in ck.save(out=None if args.in_place else args.out):+ print(f"wrote {name}")+ if args.plan:+ os.remove(args.plan)+ print("done: run diagnose.py on the result to check it")+ return 0+++if __name__ == "__main__":+ sys.exit(main())