Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.
Refactor model components and enhance functionality
HenryNdubuakuf2710bfcf95f
diff --git a/README.md b/README.mdindex 9589d31..30bb788 100644--- a/README.md+++ b/README.md@@ -121,7 +121,7 @@ needle finetune data.jsonl --epochs 10needle finetune data.jsonl --epochs 10 --generate 300 --lora-rank 16 --lora-alpha 32```-Key options: `--epochs` (default 3), `--lora-rank` (16), `--lora-alpha` (32), `--lr` (1e-4), `--batch-size` (16), `--max-len` (1024), `--val-split` (0.1), `--checkpoint <base.safetensors or .pkl>`, `--checkpoint-dir <dir>` (default `checkpoints`), `--out <adapter.safetensors or .pkl>`, `--generate <n>`, `--model <id>` (default `deepseek/deepseek-v4-flash`), and `--workers <n>` (default 8). `--generate` uses the configured OpenRouter endpoint to synthesize extra examples before training. The adapter is written to `checkpoints/needle_lora.pkl` by default. A validation loss prints each epoch from the held out split.+Key options: `--epochs` (default 3), `--layers <n>` (fine-tune the n-layer rung of the base, see below), `--lora-rank` (16), `--lora-alpha` (32), `--lr` (1e-4), `--batch-size` (16), `--max-len` (1024), `--val-split` (0.1), `--checkpoint <base.safetensors or .pkl>`, `--checkpoint-dir <dir>` (default `checkpoints`), `--out <adapter.safetensors or .pkl>`, `--generate <n>`, `--model <id>` (default `deepseek/deepseek-v4-flash`), and `--workers <n>` (default 8). `--generate` uses the configured OpenRouter endpoint to synthesize extra examples before training. The adapter is written to `checkpoints/needle_lora.pkl` by default. A validation loss prints each epoch from the held out split.Training is plain JAX and runs on any accelerator jax supports. On an NVIDIA machine install the CUDA build and the same command trains on the GPU:@@ -135,10 +135,16 @@ On Apple Silicon the `metal` extra trains on the GPU:pip install "cactus-needle[train,metal]"```+The base is Needle 3, a depth ladder: every rung from 2 layers up to the full stack is a trained model whose blocks are a nested subset of the full one. `--layers n` slices the base to its n-layer rung before training, trains that rung at full depth, and records the depth in the adapter, so `needle build` exports the same rung. A smaller rung trains and runs faster and fits a smaller device at some accuracy cost; without `--layers` the full base is fine-tuned.++```sh+needle finetune data.jsonl --epochs 10 --layers 8+```+**3. Build a tuned `.cact`.** Merge the adapter into the base and quantize. The base auto-downloads if absent.```sh-needle build checkpoints/needle2.pkl --lora checkpoints/needle_lora.safetensors --out my_needle.cact+needle build checkpoints/needle3.safetensors --lora checkpoints/needle_lora.safetensors --out my_needle.cact```Add `--bits 2` for a smaller model (by default the export follows the checkpoint's declared per-layer bit map, falling back to 4 when the checkpoint declares none), or set `NEEDLE_HF_REPO=<you>/<model>` and pass `--upload` to publish the `.cact`. The counterpart `needle download <you>/<model>/my_needle.cact` pulls a published archive on any machine, and `needle download <platform>` (e.g. `macos-arm64`) fetches that platform's engine runner.diff --git a/doc/finetuning.md b/doc/finetuning.mdindex 26119cf..d18e70d 100644--- a/doc/finetuning.md+++ b/doc/finetuning.md@@ -30,10 +30,19 @@ Train, export, load:```shneedle finetune data.jsonl --epochs 10 --out adapter.safetensors-needle build checkpoints/needle2.pkl --lora adapter.safetensors --out tuned.cact+needle build checkpoints/needle3.safetensors --lora adapter.safetensors --out tuned.cact```-Checkpoints and adapters are `.safetensors`; the Needle 2 base shipped as `checkpoints/needle2.pkl` and pickle files still load.+Checkpoints and adapters are `.safetensors`; pickle files still load. The base is `checkpoints/needle3.safetensors` (the 16-layer Needle 3); `needle3_enterprise.safetensors` is the 20-layer parent it was sliced from and fine-tunes the same way.++## Depth++Needle 3 is a depth ladder: every rung from 2 layers to the full stack was trained as a model, and a rung's blocks are a nested subset of the full stack. `--layers n` slices the base to its n-layer rung before training and fine-tunes that rung at its full depth, the adapter records the depth, and `needle build` exports the same rung (passing a different `--layers` to build is refused). Pick a rung by device budget: fewer layers is proportionally faster and smaller, at some accuracy cost. `needle build --layers n` without an adapter exports an untuned rung.++```sh+needle finetune data.jsonl --epochs 10 --layers 8 --out adapter.safetensors+needle build checkpoints/needle3.safetensors --lora adapter.safetensors --out tuned_8l.cact+``````pythonagent = needle.Needle(tools=[...], weights="tuned.cact")@@ -90,6 +99,8 @@ The confidence head. Scores are calibrated for the base model on its training miThe tokenizer. Non English text fragments into roughly 1.7 times more tokens (measured on Spanish), which taxes both quality and the 256 token window.+Audio input. A tuned `.cact` built by this package ships the text vocabulary and no speech tokenizer, so it answers text only; the base archive keeps audio.+## Troubleshooting1. Loss goes NaN within the first steps on CPU: fixed, run `pip install --upgrade cactus-needle`.diff --git a/llms.txt b/llms.txtindex a8d1c8f..cdcb381 100644--- a/llms.txt+++ b/llms.txt@@ -167,7 +167,7 @@ pip install "cactus-needle[train]"export OPENROUTER_API_KEY=sk-or-...needle generate-data --tools my_tools.json --num-samples 500 --output data.jsonl # optionalneedle finetune data.jsonl --epochs 3 --generate 300 # LoRA; auto-downloads base-needle build checkpoints/needle2.pkl --lora checkpoints/needle_lora.safetensors --out my.cact+needle build checkpoints/needle3.safetensors --lora checkpoints/needle_lora.safetensors --out my.cact```Run a tuned model - the engine is weights-agnostic:@@ -177,7 +177,7 @@ agent = needle.Needle(weights="my.cact", tools=[...])agent.run("...")```-Options: `finetune --lora-rank 16 --lora-alpha 32 --lr 1e-4 --batch-size 16 --max-len 1024 --val-split 0.1 --checkpoint <base.safetensors or .pkl> --checkpoint-dir <dir> --out <adapter.safetensors or .pkl> --generate <n> --model <id> --workers <n>`; `--model` defaults to `deepseek/deepseek-v4-flash`, `--workers` defaults to `8`, and `--checkpoint-dir` defaults to `checkpoints`. `--generate` synthesizes extra examples through the configured OpenRouter endpoint before training. `build --bits 2|4 --upload` (with `NEEDLE_HF_REPO=<you>/<model>`).+Options: `finetune --layers <n> (fine-tune the n-layer rung of the depth ladder; build exports the same rung) --lora-rank 16 --lora-alpha 32 --lr 1e-4 --batch-size 16 --max-len 1024 --val-split 0.1 --checkpoint <base.safetensors or .pkl> --checkpoint-dir <dir> --out <adapter.safetensors or .pkl> --generate <n> --model <id> --workers <n>`; `--model` defaults to `deepseek/deepseek-v4-flash`, `--workers` defaults to `8`, and `--checkpoint-dir` defaults to `checkpoints`. `--generate` synthesizes extra examples through the configured OpenRouter endpoint before training. `build --bits 2|4 --upload` (with `NEEDLE_HF_REPO=<you>/<model>`).## Playground (browser)@@ -214,7 +214,7 @@ Adapt by swapping the Literal values for your own and keeping the shapes: closed- Do not create one `Needle` per turn for a conversation; reuse the instance so context carries. New tools = new instance.- `import needle` does not import JAX; only `needle finetune`/`build` do.- `weights=` expects a `.cact` (from `needle build`), not a `.safetensors` or `.pkl` checkpoint.-- Checkpoints and adapters are `.safetensors`; only the Needle 2 base shipped as a `.pkl`, and both formats load.+- Checkpoints and adapters are `.safetensors`; pickle checkpoints still load. The base is `checkpoints/needle3.safetensors` (16 layers); `needle3_enterprise.safetensors` is the 20-layer parent.- The engine cannot unload weights: once a tuned `.cact` is bound, constructing or calling a base-model agent raises instead of silently answering with those weights. Construct agents that want the base model before any tuned one, or run them in separate processes.## Telemetrydiff --git a/needle/__init__.py b/needle/__init__.pyindex 29c5565..c98e035 100644--- a/needle/__init__.py+++ b/needle/__init__.py@@ -147,6 +147,9 @@ def _lib(generation=2):lib.needle_reset.restype = Nonelib.needle_load.argtypes = [ctypes.c_char_p, ctypes.c_uint64]lib.needle_load.restype = ctypes.c_int+ if hasattr(lib, "needle_set_max_calls"):+ lib.needle_set_max_calls.argtypes = [ctypes.c_int]+ lib.needle_set_max_calls.restype = None_lib_handles[generation] = libreturn _lib_handles[generation]@@ -226,6 +229,13 @@ class Needle:payload = _prepare_audio(audio, audio_format, sample_rate, channels)return self._complete(text, max_new_tokens, payload)+ def _set_max_calls(self, max_calls: int) -> None:+ if self._worker is not None:+ return+ lib = _lib(self._generation)+ if hasattr(lib, "needle_set_max_calls"):+ lib.needle_set_max_calls(int(max_calls))+def _complete(self, text: str, max_new_tokens: int = 256,audio=None, ground: bool = True) -> dict:if audio is not None and self._generation < 3:@@ -548,7 +558,11 @@ def extract(text: str, schema: type | dict, system: str | None = None,"generation": generation})agent = Needle(tools=[schema], system=system, weights=selected)try:- response = agent._complete(text, max_new_tokens)+ agent._set_max_calls(1)+ try:+ response = agent._complete(text, max_new_tokens)+ finally:+ agent._set_max_calls(0)finally:agent.close()calls = response.get("function_calls") or []diff --git a/needle/cli.py b/needle/cli.pyindex 93b4fa5..dc3a9d7 100644--- a/needle/cli.py+++ b/needle/cli.py@@ -155,6 +155,9 @@ def main():"--qat-bits", choices=["auto", "none", "2", "4"], default="auto",help="LoRA training numerics: auto matches the checkpoint export scheme ""(default), or force none/2/4")+ p.add_argument("--layers", type=int, default=None,+ help="Fine-tune the N-layer rung of the base (2..L); the adapter "+ "and the built .cact keep that depth")p = sub.add_parser("generate-data")p.add_argument("--tools", type=str, default=None, help="Tool schemas JSON to seed generation")@@ -173,6 +176,9 @@ def main():p.add_argument("--out", type=str, default=None, help="Output .cact path")p.add_argument("--upload", action="store_true", help="Push the .cact to $NEEDLE_HF_REPO")p.add_argument("--bits", type=str, default=None, choices=["2", "4"])+ p.add_argument("--layers", type=int, default=None,+ help="Export the N-layer rung of the checkpoint (an adapter "+ "trained with --layers sets this itself)")p = sub.add_parser("download")p.add_argument("spec", type=str,diff --git a/needle/model/architecture.py b/needle/model/architecture.pyindex 8d35d51..21c289d 100644--- a/needle/model/architecture.py+++ b/needle/model/architecture.py@@ -1,5 +1,5 @@import math-from dataclasses import dataclass+from dataclasses import dataclass, replaceimport numpy as npimport jax@@ -8,17 +8,7 @@ import jax.nn.initializers as jinitimport flax.linen as nnfrom . import quantize as _quantize-from .quantize import fake_quant_act---def head_dims(config):- legacy = (getattr(config, "attn_dim", 0) or config.d_model) // config.num_heads- qk = getattr(config, "qk_head_dim", 0) or legacy- v = getattr(config, "v_head_dim", 0) or legacy- return qk, v---+from .quantize import fake_quant_act, head_weightdef _aq(x, quant):if quant is False:return x@@ -35,14 +25,6 @@ def residual_init(num_layers):DTYPE_MAP = {"float32": jnp.float32, "bfloat16": jnp.bfloat16, "float16": jnp.float16}-PRESETS = {- "needle": dict(d_model=768, num_heads=12, num_kv_heads=6, num_layers=27,- engram_layers=(2, 15)),-}-PRESETS["base"] = dict(d_model=512, num_heads=8, num_kv_heads=4, num_layers=27,- engram_layers=(2, 15))--class ZCRMSNorm(nn.Module):epsilon: float = 1e-6dtype: jnp.dtype = jnp.bfloat16@@ -57,23 +39,45 @@ class ZCRMSNorm(nn.Module):@dataclassclass TransformerConfig:- vocab_size: int = 8192- d_model: int = 512- attn_dim: int = 0- num_heads: int = 8- num_kv_heads: int = 4- num_layers: int = 12- max_seq_len: int = 2048+ vocab_size: int = 16384+ d_model: int = 768+ num_heads: int = 12+ num_kv_heads: int = 2+ num_layers: int = 20+ qk_head_dim: int = 48+ v_head_dim: int = 64+ max_seq_len: int = 4096pad_token_id: int = 0- contrastive_dim: int = 128+ embedding_dim: int = 128+ embedding_probes: int = 4+ embedding_queries: int = 4+ confidence_probes: int = 4+ confidence_queries: int = 4+ router_probes: int = 4+ router_queries: int = 4rope_theta: float = 100000.0dtype: str = "bfloat16"flash: bool = Trueengram_orders: tuple = (2, 3)engram_heads: int = 0- engram_slots: int = 8192- engram_layers: tuple = (2, 15)+ engram_slots: int = 18432+ engram_seed_heads: int = 0+ engram_layers: tuple = (3, 7, 11, 15, 19)+ global_layers: tuple = (4, 9, 14, 19)+ sliding_window: int = 1024+ ladder_depths: tuple = ()+ ladder_sample: bool = False+ ladder_widths: tuple = ()mhc_lanes: int = 4+ qkv_conv_taps: int = 3+ out_vocab: int = 0+ image_code_offset: int = 0+ image_sem_codes: int = 0+ image_sem_dim: int = 0+ image_tex_books: int = 0+ image_tex_codes: int = 0+ image_tex_stages: int = 0+ image_tex_dim: int = 0kv_window: int = 0kv_bits: int = 8act_bits: int = 8@@ -86,14 +90,286 @@ class TransformerConfig:for k, v in kwargs.items():if k in valid:setattr(self, k, v)- self.attn_dim = self.attn_dim or self.d_model+ legacy_attn = kwargs.get("attn_dim", 0)+ if legacy_attn:+ legacy_head = legacy_attn // self.num_heads+ if "qk_head_dim" not in kwargs:+ self.qk_head_dim = legacy_head+ if "v_head_dim" not in kwargs:+ self.v_head_dim = legacy_head+ if "sliding_window" not in kwargs:+ self.sliding_window = 0+ if "global_layers" not in kwargs:+ self.global_layers = ()self.engram_layers = tuple(self.engram_layers)+ self.global_layers = tuple(self.global_layers)+ self.ladder_depths = tuple(self.ladder_depths)+ self.ladder_widths = tuple(self.ladder_widths)++ @classmethod+ def from_saved(cls, saved):+ if not isinstance(saved, dict):+ saved = dict(saved.__dict__)+ allowed = {f.name for f in cls.__dataclass_fields__.values()}+ saved = {k: v for k, v in saved.items() if k in allowed}+ for key, off in (("qk_head_dim", 0), ("v_head_dim", 0),+ ("sliding_window", 0), ("global_layers", ()),+ ("ladder_depths", ()), ("ladder_sample", False),+ ("ladder_widths", ()), ("engram_seed_heads", 0),+ ("qkv_conv_taps", 0), ("out_vocab", 0),+ ("image_code_offset", 0), ("image_sem_codes", 0),+ ("image_sem_dim", 0), ("image_tex_books", 0),+ ("image_tex_codes", 0), ("image_tex_stages", 0),+ ("image_tex_dim", 0)):+ saved.setdefault(key, off)+ return cls(**saved)@propertydef jax_dtype(self):return DTYPE_MAP[self.dtype]+def head_dims(config):+ legacy = (getattr(config, "attn_dim", 0) or config.d_model) // config.num_heads+ qk = getattr(config, "qk_head_dim", 0) or legacy+ v = getattr(config, "v_head_dim", 0) or legacy+ return qk, v+++def _ladder_layer_order(num_layers):+ if num_layers < 1:+ raise ValueError("models require at least one layer")+ if num_layers == 1:+ return (0,)+ selected = [0, num_layers - 1]+ order = list(selected)+ while len(order) < num_layers:+ selected.sort()+ _gap, left, right = max(+ ((right - left, left, right)+ for left, right in zip(selected, selected[1:])+ if right - left > 1),+ key=lambda item: (item[0], -item[1]),+ )+ candidate = (left + right) // 2+ selected.append(candidate)+ order.append(candidate)+ return tuple(order)+++def ladder_layer_indices(num_layers, depth):+ """Return the stable, nested original-layer indices for a depth rung.++ Every deployable rung keeps both endpoint blocks. Interior blocks are+ added one at a time by bisecting the largest remaining gap, producing+ deterministic nested subnetworks with spatially balanced coverage.+ """+ if not 2 <= depth <= num_layers:+ raise ValueError(+ f"ladder depth must be in [2, {num_layers}], got {depth}")+ return tuple(sorted(_ladder_layer_order(num_layers)[:depth]))+++def ladder_layer_ranks(num_layers):+ """Map each original block to its selection rank in the nested ladder."""+ order = _ladder_layer_order(num_layers)+ ranks = [0] * num_layers+ for rank, layer in enumerate(order):+ ranks[layer] = rank+ return tuple(ranks)+++def ladder_config(config, depth):+ assert depth < config.num_layers+ assert tuple(config.engram_layers) == tuple(sorted(config.engram_layers))+ selected = ladder_layer_indices(config.num_layers, depth)+ remap = {layer: i for i, layer in enumerate(selected)}+ fields = dict(config.__dict__)+ fields["num_layers"] = depth+ fields["global_layers"] = tuple(+ remap[l] for l in config.global_layers if l in remap)+ fields["engram_layers"] = tuple(+ remap[l] for l in config.engram_layers if l in remap)+ fields["ladder_depths"] = ()+ fields["ladder_sample"] = False+ return TransformerConfig(**fields)+++def ladder_row_keep(config, exit_depth=None):+ if exit_depth is None:+ return None+ ranks = jnp.asarray(ladder_layer_ranks(config.num_layers), jnp.int32)+ return jnp.concatenate([jnp.ones((1,), jnp.float32),+ (ranks < exit_depth).astype(jnp.float32)])+++def ladder_slice(params, config, depth):+ selected = ladder_layer_indices(config.num_layers, depth)+ selected_set = set(selected)+ selected_array = jnp.asarray(selected, dtype=jnp.int32)+ rows = jnp.asarray((0, *(layer + 1 for layer in selected)), jnp.int32)+ out = {}+ for k, v in params.items():+ if k == "stack":+ out[k] = {sk: (sv if sk == "final_norm"+ else jax.tree.map(+ lambda a: jnp.take(a, selected_array, axis=0), sv))+ for sk, sv in v.items()}+ elif k in HEAD_KEYS:+ def take_rows(name, axis):+ return jax.tree.map(lambda a: jnp.take(a, rows, axis=axis), v[name])+ out[k] = {**v, "probes": take_rows("probes", 0),+ "gain": take_rows("gain", 0), "row_bias": take_rows("row_bias", 1)}+ elif k.startswith("engrams_"):+ site = int(k.rsplit("_", 1)[1])+ if config.engram_layers[site] in selected_set:+ new_site = sum(+ layer in selected_set for layer in config.engram_layers[:site])+ out[f"engrams_{new_site}"] = v+ else:+ out[k] = v+ return out+++def parse_ladder_depths(raw):+ if ".." in raw:+ lo, hi = raw.split("..")+ return tuple(range(int(lo), int(hi) + 1)), True+ return tuple(int(v) for v in raw.split(",") if v), False+++def parse_ladder_widths(raw):+ return tuple(int(v) for v in raw.split(",") if v)+++def width_config(config, w):+ assert 2 * w == config.d_model+ assert config.d_model & (config.d_model - 1) == 0+ assert config.num_heads % 2 == 0+ assert _hada_blocks(w)[1] == _hada_blocks(config.d_model)[1]+ orders, heads, _ = engram_geometry(config)+ if config.engram_layers:+ assert heads % 2 == 0+ fields = dict(config.__dict__)+ fields.update(+ d_model=w,+ num_heads=config.num_heads // 2,+ engram_seed_heads=getattr(config, "engram_seed_heads", 0) or heads,+ ladder_depths=(),+ ladder_sample=False,+ ladder_widths=(),+ )+ return TransformerConfig(**fields)+++def _parameter_path(path):+ """Return a stable parameter path through optional Flax metadata nodes."""+ parts = []+ for entry in path:+ if hasattr(entry, "key"):+ parts.append(str(entry.key))+ elif getattr(entry, "name", None) == "value":+ continue+ elif hasattr(entry, "idx"):+ parts.append(str(entry.idx))+ else:+ parts.append(str(entry))+ return "/".join(parts)+++def width_slice(params, config, w):+ D = config.d_model+ orders, heads, sub_dim = engram_geometry(config)+ nh = config.num_heads // 2+ qk_hd, v_hd = head_dims(config)+ q_cols, o_rows = nh * qk_hd, nh * v_hd+ ba_c = _hada_blocks(1 << (w - 1).bit_length())[0]+ kept = (np.concatenate([oi * heads + np.arange(heads // 2)+ for oi in range(len(orders))])+ if config.engram_layers else None)+ kv_rows = ((kept[:, None] * sub_dim + np.arange(sub_dim)).ravel()+ if config.engram_layers else None)+ lane_rows = (np.arange(config.mhc_lanes)[:, None] * D + np.arange(w)).ravel()++ def cut(path, a):+ p = _parameter_path(path)+ if p.startswith("ab_scales"):+ return a+ if "engrams_" in p:+ if p.endswith("/embedding"):+ return a[kept]+ if p.endswith("key_proj/kernel") or p.endswith("value_proj/kernel"):+ return a[kv_rows][:, :w]+ if p.endswith("/taps"):+ return a[..., :w]+ return a+ if ("embedding_head" in p or "confidence_head" in p+ or "router_head" in p):+ if p.endswith("/probes") or p.endswith("/query"):+ return a[..., :w]+ if p.endswith("proj/kernel"):+ return a.reshape(-1, D, a.shape[-1])[:, :w].reshape(-1, a.shape[-1])+ return a+ if "mhc_phi" in p:+ return a[:, lane_rows]+ if "hadamard_mlp" in p:+ last = p.rsplit("/", 1)[1]+ if last in ("w1a", "w2a", "w3a"):+ return a[..., :ba_c, :ba_c]+ if last in ("d1", "d2", "b2", "d3", "d4"):+ return a[..., :w]+ if last == "cond_v":+ return a[..., :w, :]+ if last == "cond_u":+ return a[..., :w]+ return a+ if p.endswith("q_proj/kernel"):+ return a[..., :w, :q_cols]+ if p.endswith("gate_proj/kernel"):+ return a[..., :w, :o_rows]+ if p.endswith("k_proj/kernel") or p.endswith("v_proj/kernel"):+ return a[..., :w, :]+ if p.endswith("out_proj/kernel"):+ return a[..., :o_rows, :w]+ if p.endswith("/q_taps"):+ return a[..., :q_cols]+ if p.endswith("/k_taps") or p.endswith("/v_taps"):+ return a+ if p.endswith("q_norm/scale") or p.endswith("k_norm/scale"):+ return a+ if p.endswith("/scale"):+ return a[..., :w]+ if p == "embedding/embedding":+ return a[..., :w]+ if p.endswith("image_projection/kernel"):+ return a[..., :w]+ return a++ return jax.tree_util.tree_map_with_path(cut, params)+++def ladder_pool(config, weight):+ pool = []+ for depth in (d for d in config.ladder_depths if d < config.num_layers):+ dcfg = ladder_config(config, depth)+ pool.append((depth, config.d_model,+ SimpleAttentionNetwork(dcfg).apply,+ (lambda p, _d=depth: ladder_slice(p, config, _d)),+ weight * depth / config.num_layers))+ for w in config.ladder_widths:+ pool.append((depth, w,+ SimpleAttentionNetwork(width_config(dcfg, w)).apply,+ (lambda p, _d=depth, _dc=dcfg, _w=w:+ width_slice(ladder_slice(p, config, _d), _dc, _w)),+ weight * (depth / config.num_layers) * (w / config.d_model)))+ for w in config.ladder_widths:+ pool.append((config.num_layers, w,+ SimpleAttentionNetwork(width_config(config, w)).apply,+ (lambda p, _w=w: width_slice(p, config, _w)),+ weight * w / config.d_model))+ return pool++def precompute_rope_freqs(head_dim, seq_len, theta=10000.0):freqs = 1.0 / (theta ** (jnp.arange(0, head_dim, 2).astype(jnp.float32) / head_dim))t = jnp.arange(seq_len).astype(jnp.float32)@@ -143,12 +419,13 @@ def _mask_diag(mask, offset):return jnp.pad(d, ((0, 0), (offset, 0)))-def engram_indices(tokens, orders, heads, slots):+def engram_indices(tokens, orders, heads, slots, seed_heads=0):u = tokens.astype(jnp.uint32)+ stride = seed_heads or headsidx = []for oi, order in enumerate(orders):for h in range(heads):- seed = (_ENGRAM_SEED * (oi * heads + h + 1)) & 0xFFFFFFFF+ seed = (_ENGRAM_SEED * (oi * stride + h + 1)) & 0xFFFFFFFFacc = jnp.full_like(u, jnp.uint32(seed))for j in range(order):acc = (acc ^ _shift_right(u, j)) * jnp.uint32(_ENGRAM_PRIME)@@ -213,23 +490,46 @@ class MultiHeadAttention(nn.Module):num_layers: intdtype: jnp.dtype = jnp.bfloat16flash: bool = True- attn_dim: int = 0+ qk_head_dim: int = 0+ v_head_dim: int = 0+ qkv_conv_taps: int = 0@nn.compactdef __call__(self, x, mask=None, rope=None, quant=False):- attn_dim = self.attn_dim or self.d_model- head_dim = attn_dim // self.num_heads- kv_dim = self.num_kv_heads * head_dim+ qk_hd = self.qk_head_dim or self.d_model // self.num_heads+ v_hd = self.v_head_dim or self.d_model // self.num_heads+ out_dim = self.num_heads * v_hdB = x.shape[0]x = _aq(x, quant)- q = nn.Dense(attn_dim, dtype=self.dtype, use_bias=False, kernel_init=default_init(), name="q_proj")(x)- k = nn.Dense(kv_dim, dtype=self.dtype, use_bias=False, kernel_init=default_init(), name="k_proj")(x)- v = nn.Dense(kv_dim, dtype=self.dtype, use_bias=False, kernel_init=default_init(), name="v_proj")(x)-- q = q.reshape(B, -1, self.num_heads, head_dim).transpose(0, 2, 1, 3)- k = k.reshape(B, -1, self.num_kv_heads, head_dim).transpose(0, 2, 1, 3)- v = v.reshape(B, -1, self.num_kv_heads, head_dim).transpose(0, 2, 1, 3)+ q = nn.Dense(self.num_heads * qk_hd, dtype=self.dtype, use_bias=False, kernel_init=default_init(), name="q_proj")(x)+ k = nn.Dense(self.num_kv_heads * qk_hd, dtype=self.dtype, use_bias=False, kernel_init=default_init(), name="k_proj")(x)+ v = nn.Dense(self.num_kv_heads * v_hd, dtype=self.dtype, use_bias=False, kernel_init=default_init(), name="v_proj")(x)++ if self.qkv_conv_taps:+ n_taps = self.qkv_conv_taps+ qt = self.param("q_taps", _conv_identity_init,+ (n_taps, self.num_heads * qk_hd)).astype(self.dtype)+ kt = self.param("k_taps", _conv_identity_init,+ (n_taps, self.num_kv_heads * qk_hd)).astype(self.dtype)+ vt = self.param("v_taps", _conv_identity_init,+ (n_taps, self.num_kv_heads * v_hd)).astype(self.dtype)+ tap_ok = [None] + [+ (_mask_diag(mask, j)[..., None].astype(q.dtype)+ if mask is not None else None)+ for j in range(1, n_taps)]++ def tapped(z, j):+ shifted = _shift_right(z, j)+ return shifted if tap_ok[j] is None else shifted * tap_ok[j]++ q = sum(qt[j] * tapped(q, j) for j in range(n_taps))+ k = sum(kt[j] * tapped(k, j) for j in range(n_taps))+ v = sum(vt[j] * tapped(v, j) for j in range(n_taps))++ q = q.reshape(B, -1, self.num_heads, qk_hd).transpose(0, 2, 1, 3)+ k = k.reshape(B, -1, self.num_kv_heads, qk_hd).transpose(0, 2, 1, 3)+ v = v.reshape(B, -1, self.num_kv_heads, v_hd).transpose(0, 2, 1, 3)q = ZCRMSNorm(dtype=self.dtype, name="q_norm")(q)k = ZCRMSNorm(dtype=self.dtype, name="k_norm")(k)@@ -239,27 +539,36 @@ class MultiHeadAttention(nn.Module):q = apply_rope(q, cos, sin)k = apply_rope(k, cos, sin)+ q = _quantize.maybe_quant_query(q, quant)k = _quantize.maybe_quant_kv(k, quant)v = _quantize.maybe_quant_kv(v, quant)if self.flash:+ fused_hd = max(qk_hd, v_hd)+ qf = q * jnp.asarray(math.sqrt(fused_hd / qk_hd), q.dtype)+ pad_qk = fused_hd - qk_hd+ if pad_qk:+ qf = jnp.pad(qf, ((0, 0), (0, 0), (0, 0), (0, pad_qk)))+ k = jnp.pad(k, ((0, 0), (0, 0), (0, 0), (0, pad_qk)))+ pad_v = fused_hd - v_hd+ vf = jnp.pad(v, ((0, 0), (0, 0), (0, 0), (0, pad_v))) if pad_v else vimpl = ("cudnn" if jax.default_backend() == "gpu"and q.dtype in (jnp.bfloat16, jnp.float16) else None)out = jax.nn.dot_product_attention(- q.transpose(0, 2, 1, 3),+ qf.transpose(0, 2, 1, 3),k.transpose(0, 2, 1, 3),- v.transpose(0, 2, 1, 3),+ vf.transpose(0, 2, 1, 3),mask=mask,implementation=impl,)- out = out.reshape(B, -1, attn_dim)+ out = out[..., :v_hd].reshape(B, -1, out_dim)else:repeats = self.num_heads // self.num_kv_headsif repeats > 1:k = jnp.repeat(k, repeats, axis=1)v = jnp.repeat(v, repeats, axis=1)- scale = jnp.sqrt(jnp.float32(head_dim))+ scale = jnp.sqrt(jnp.float32(qk_hd))attn_weights = jnp.matmul(q, k.transpose(0, 1, 3, 2)) / scaleif mask is not None:@@ -268,10 +577,10 @@ class MultiHeadAttention(nn.Module):attn_weights = nn.softmax(attn_weights, axis=-1)out = jnp.matmul(attn_weights, v)- out = out.transpose(0, 2, 1, 3).reshape(B, -1, attn_dim)+ out = out.transpose(0, 2, 1, 3).reshape(B, -1, out_dim)out = out * nn.sigmoid(- nn.Dense(attn_dim, dtype=self.dtype, use_bias=False,+ nn.Dense(out_dim, dtype=self.dtype, use_bias=False,kernel_init=default_init(), name="gate_proj")(x))out = _aq(out, quant)return nn.Dense(self.d_model, dtype=self.dtype, use_bias=False, kernel_init=residual_init(self.num_layers), name="out_proj")(out)@@ -284,22 +593,70 @@ def _walsh_matrix(n):return jnp.asarray(H / np.sqrt(n))+HADA_COND_RANK = 8+_HADA_PERM_SEEDS = (11, 13)+++def _hada_blocks(n):+ b = 1 << ((n - 1).bit_length() // 2)+ return b, n // b+++def _hada_perms(n, split=False):+ if split:+ h = n // 2+ return tuple(jnp.asarray(np.concatenate(+ [np.random.RandomState(s).permutation(h),+ h + np.random.RandomState(s + 977).permutation(h)]))+ for s in _HADA_PERM_SEEDS)+ return tuple(jnp.asarray(np.random.RandomState(s).permutation(n))+ for s in _HADA_PERM_SEEDS)+++def _kron_apply(z, a, b):+ lead = z.shape[:-1]+ z = z.reshape(*lead, a.shape[0], b.shape[0])+ z = jnp.einsum("...ij,ik,jl->...kl", z, a, b)+ return z.reshape(*lead, a.shape[0] * b.shape[0])+++def _walsh_init(size):+ w = _walsh_matrix(size)+ return lambda key, shape: jnp.asarray(w)++class HadamardMLP(nn.Module):d_model: intdtype: jnp.dtype = jnp.bfloat16+ split_perms: bool = False@nn.compactdef __call__(self, x):n = 1 << (self.d_model - 1).bit_length()- H = _walsh_matrix(n).astype(self.dtype)+ ba, bb = _hada_blocks(n)+ p1, p2 = _hada_perms(n, self.split_perms)+ factors = [+ (self.param(f"w{s}a", _walsh_init(ba), (ba, ba)).astype(self.dtype),+ self.param(f"w{s}b", _walsh_init(bb), (bb, bb)).astype(self.dtype))+ for s in (1, 2, 3)+ ]d1 = self.param("d1", jinit.ones, (n,)).astype(self.dtype)d2 = self.param("d2", jinit.ones, (n,)).astype(self.dtype)- d3 = self.param("d3", jinit.constant(0.02), (n,)).astype(self.dtype)+ b2 = self.param("b2", jinit.zeros, (n,)).astype(self.dtype)+ d3 = self.param("d3", jinit.ones, (n,)).astype(self.dtype)+ d4 = self.param("d4", jinit.constant(0.02), (n,)).astype(self.dtype)+ cond_v = self.param("cond_v", default_init(),+ (self.d_model, HADA_COND_RANK)).astype(self.dtype)+ cond_u = self.param("cond_u", jinit.zeros,+ (HADA_COND_RANK, n)).astype(self.dtype)++ cond = 1 + nn.softmax(x @ cond_v, axis=-1) @ cond_upad = n - self.d_modelz = jnp.pad(x, ((0, 0), (0, 0), (0, pad))) if pad else x- z = (d1 * z) @ H- z = nn.silu(d2 * z) @ H- return (d3 * z)[..., : self.d_model]+ z = _kron_apply(d1 * z, *factors[0])[..., p1]+ z = _kron_apply(nn.silu(d2 * cond * z + b2), *factors[1])[..., p2]+ z = _kron_apply(d3 * z, *factors[2])+ return (d4 * z)[..., : self.d_model]class Block(nn.Module):@@ -309,7 +666,10 @@ class Block(nn.Module):num_layers: intdtype: jnp.dtype = jnp.bfloat16flash: bool = True- attn_dim: int = 0+ qk_head_dim: int = 0+ v_head_dim: int = 0+ qkv_conv_taps: int = 0+ hada_split: bool = Falsedef _gate(self, name):return nn.sigmoid(self.param(name, jinit.zeros, ())).astype(self.dtype)@@ -326,14 +686,17 @@ class Block(nn.Module):skip = xx = ZCRMSNorm(dtype=self.dtype)(x)x = MultiHeadAttention(self.num_heads, self.num_kv_heads, self.d_model, self.num_layers,- self.dtype, self.flash, attn_dim=self.attn_dim,+ self.dtype, self.flash, qk_head_dim=self.qk_head_dim,+ v_head_dim=self.v_head_dim,+ qkv_conv_taps=self.qkv_conv_taps,name="self_attn")(x, mask=mask, rope=rope, quant=quant)x = ZCRMSNorm(dtype=self.dtype, name="post_attn_norm")(x)x = skip + self._gate("attn_gate") * xskip = xx = ZCRMSNorm(dtype=self.dtype, name="pre_hada_norm")(x)- x = HadamardMLP(self.d_model, self.dtype, name="hadamard_mlp")(x)+ x = HadamardMLP(self.d_model, self.dtype, split_perms=self.hada_split,+ name="hadamard_mlp")(x)return skip + x@@ -345,30 +708,71 @@ class _ScanBody(nn.Module):dtype: jnp.dtype = jnp.bfloat16flash: bool = Truecollect_hidden: bool = False- attn_dim: int = 0+ qk_head_dim: int = 0+ v_head_dim: int = 0+ qkv_conv_taps: int = 0+ hada_split: bool = False+ capture_subnetwork: bool = False+ subnetwork_only: bool = False@nn.compact- def __call__(self, x, xs, mask, rope, quant, engram_kv):- site_flags, hc = xs+ def __call__(self, carry, xs, mask, local_mask, rope, quant, engram_kv,+ exit_depth):+ if self.capture_subnetwork:+ x, sub_x = carry+ else:+ x = carry+ site_flags, gflag, hc, layer_rank = xs+ if local_mask is not None:+ mask = jnp.where(gflag > 0, mask, local_mask)block = Block(self.num_heads, self.num_kv_heads, self.d_model, self.num_layers,- self.dtype, self.flash, attn_dim=self.attn_dim,- name="block",+ self.dtype, self.flash, qk_head_dim=self.qk_head_dim,+ v_head_dim=self.v_head_dim, qkv_conv_taps=self.qkv_conv_taps,+ hada_split=self.hada_split, name="block",)- B, T, n, C = x.shape- xf = x.astype(jnp.float32)- nx = _rms_unit(x.reshape(B, T, n * C))- hpre = nn.sigmoid(hc["a_pre"] * (nx @ hc["phi_pre"].astype(jnp.float32))- + hc["b_pre"] + hc["pre_off"])- u = jnp.einsum("btn,btnc->btc", hpre, xf).astype(self.dtype)- y = block(u, mask=mask, rope=rope, quant=quant,- engram_kv=engram_kv, site_flags=site_flags) - u- hpost = 2 * nn.sigmoid(hc["a_post"] * (nx @ hc["phi_post"].astype(jnp.float32))- + hc["b_post"] + hc["post_off"])- res = nx @ hc["phi_res"].astype(jnp.float32)- hres = _sinkhorn(hc["a_res"] * res.reshape(B, T, n, n) + hc["b_res"])- new_x = (jnp.einsum("btij,btjc->btic", hres, xf)- + hpost[..., None] * y.astype(jnp.float32)[:, :, None, :]).astype(self.dtype)+ def advance(stream, pre_off, post_off, block_module):+ B, T, n, C = stream.shape+ xf = stream.astype(jnp.float32)+ nx = _aq(_rms_unit(stream.reshape(B, T, n * C)), quant)+ hpre = nn.sigmoid(+ hc["a_pre"] * (nx @ hc["phi_pre"].astype(jnp.float32))+ + hc["b_pre"] + pre_off)+ u = jnp.einsum("btn,btnc->btc", hpre, xf).astype(self.dtype)+ y = block_module(u, mask=mask, rope=rope, quant=quant,+ engram_kv=engram_kv, site_flags=site_flags) - u+ hpost = 2 * nn.sigmoid(+ hc["a_post"] * (nx @ hc["phi_post"].astype(jnp.float32))+ + hc["b_post"] + post_off)+ res = nx @ hc["phi_res"].astype(jnp.float32)+ hres = _sinkhorn(+ hc["a_res"] * res.reshape(B, T, n, n) + hc["b_res"])+ return (jnp.einsum("btij,btjc->btic", hres, xf)+ + hpost[..., None]+ * y.astype(jnp.float32)[:, :, None, :]).astype(self.dtype)++ if self.subnetwork_only:+ new_x = nn.cond(+ layer_rank < exit_depth,+ lambda mdl, stream: advance(+ stream, hc["sub_pre_off"], hc["sub_post_off"], mdl),+ lambda _mdl, stream: stream,+ block,+ x,+ )+ else:+ new_x = advance(x, hc["pre_off"], hc["post_off"], block)+ if self.capture_subnetwork:+ sub_x = nn.cond(+ layer_rank < exit_depth,+ lambda mdl, stream: advance(+ stream, hc["sub_pre_off"], hc["sub_post_off"], mdl),+ lambda _mdl, stream: stream,+ block,+ sub_x,+ )+ out = jnp.mean(sub_x, axis=2) if self.collect_hidden else None+ return (new_x, sub_x), outout = jnp.mean(new_x, axis=2) if self.collect_hidden else Nonereturn new_x, out@@ -377,8 +781,9 @@ class Stack(nn.Module):config: TransformerConfig@nn.compact- def __call__(self, x, mask=None, rope=None, engram_kv=None, collect_hidden=False,- quant=False):+ def __call__(self, x, mask=None, rope=None, engram_kv=None,+ collect_hidden=False, quant=False, exit_depth=None,+ subnetwork_only=False):cfg = self.configdt = cfg.jax_dtypex = x.astype(dt)@@ -390,6 +795,15 @@ class Stack(nn.Module):flags[layer, s] = 1.0site_flags = jnp.asarray(flags)+ local_mask, gflags = None, jnp.zeros((cfg.num_layers,), jnp.float32)+ if cfg.sliding_window and mask is not None:+ T = x.shape[1]+ pos = jnp.arange(T)+ band = ((pos[:, None] - pos[None, :]) < cfg.sliding_window)[None, None]+ local_mask = mask & band+ gflags = jnp.asarray([1.0 if i in cfg.global_layers else 0.0+ for i in range(cfg.num_layers)])+n, L, nC = cfg.mhc_lanes, cfg.num_layers, cfg.mhc_lanes * cfg.d_modellane = np.eye(n, dtype=np.float32)[np.arange(L) % n]hc = {@@ -406,73 +820,188 @@ class Stack(nn.Module):"post_off": jnp.asarray(-4 * (1 - lane)),}x = jnp.broadcast_to(x[:, :, None, :], (*x.shape[:2], n, x.shape[-1]))+ if subnetwork_only and exit_depth is None:+ raise ValueError("subnetwork_only requires exit_depth")+ capture_subnetwork = exit_depth is not None and not subnetwork_only+ if capture_subnetwork:+ carry = (x, x)+ else:+ carry = x++ layer_ranks = jnp.asarray(ladder_layer_ranks(L), dtype=jnp.int32)+ if exit_depth is not None:+ active = layer_ranks < exit_depth+ sub_positions = jnp.cumsum(active.astype(jnp.int32)) - 1+ sub_lane = jax.nn.one_hot(+ jnp.mod(sub_positions, n), n, dtype=jnp.float32)+ hc["sub_pre_off"] = 8 * sub_lane - 4+ hc["sub_post_off"] = -4 * (1 - sub_lane)+ else:+ sub_positions = jnp.zeros((L,), dtype=jnp.int32)+ hc["sub_pre_off"] = hc["pre_off"]+ hc["sub_post_off"] = hc["post_off"]ScanBlock = nn.scan(nn.remat(_ScanBody) if cfg.remat else _ScanBody,variable_axes={"params": 0},split_rngs={"params": True},+ metadata_params={nn.PARTITION_NAME: "layers"},length=cfg.num_layers,- in_axes=(0, nn.broadcast, nn.broadcast, nn.broadcast, nn.broadcast),unroll=cfg.scan_unroll,+ in_axes=(0, nn.broadcast, nn.broadcast, nn.broadcast, nn.broadcast,+ nn.broadcast, nn.broadcast),)- x, hidden = ScanBlock(+ carry, hidden = ScanBlock(cfg.num_heads, cfg.num_kv_heads, cfg.d_model, cfg.num_layers, dt,- cfg.flash, collect_hidden, attn_dim=cfg.attn_dim,+ cfg.flash, collect_hidden, qk_head_dim=cfg.qk_head_dim,+ v_head_dim=cfg.v_head_dim,+ qkv_conv_taps=getattr(cfg, "qkv_conv_taps", 0),+ hada_split=bool(getattr(cfg, "ladder_widths", ())),+ capture_subnetwork=capture_subnetwork,+ subnetwork_only=subnetwork_only,name="layers",- )(x, (site_flags, hc), mask, rope, quant, engram_kv)+ )(carry, (site_flags, gflags, hc, layer_ranks),+ mask, local_mask, rope, quant, engram_kv,+ jnp.int32(0) if exit_depth is None else exit_depth)+ if capture_subnetwork:+ x, sub_x = carry+ else:+ x = carryx = jnp.mean(x, axis=2)- x = ZCRMSNorm(dtype=dt, name="final_norm")(x)+ final_norm = ZCRMSNorm(dtype=dt, name="final_norm")+ if capture_subnetwork:+ sub_x = jnp.mean(sub_x, axis=2)+ both = final_norm(jnp.concatenate((x, sub_x), axis=0))+ x, sub_x = jnp.split(both, 2, axis=0)+ return x, hidden, sub_x+ x = final_norm(x)return x, hidden-def probe_pool(cells, probes, keep=None, dtype=jnp.bfloat16):- b, t, l, d = cells.shape- cells = cells.reshape(b, t * l, d)- if keep is not None:- keep = jnp.repeat(keep, l, axis=1)- scores = jnp.einsum("bcd,kd->bkc", cells.astype(jnp.float32),+def probe_pool(cells, keep, probes, gain, query, bias, row_keep=None,+ dtype=jnp.bfloat16):+ cf = cells.astype(jnp.float32)+ b, t, l1, d = cf.shape+ k, q = probes.shape[1], query.shape[0]+ scores = jnp.einsum("btld,lkd->blkt", cf,probes.astype(jnp.float32)) / math.sqrt(d)if keep is not None:- scores = jnp.where(keep[:, None, :] > 0, scores, -jnp.inf)- w = jax.nn.softmax(scores, axis=-1)- return jnp.einsum("bkc,bcd->bkd", w, cells.astype(jnp.float32)- ).reshape(b, -1).astype(dtype)+ scores = jnp.where(keep[:, None, None, :] > 0, scores, -jnp.inf)+ r = jnp.einsum("blkt,btld->blkd", jax.nn.softmax(scores, axis=-1), cf)+ r = _rms_unit(r) * gain.astype(jnp.float32)[None, :, :, None]+ u = (jnp.einsum("blkd,qd->bqlk", r, query.astype(jnp.float32))+ / math.sqrt(d) + bias.astype(jnp.float32)[None])+ if row_keep is not None:+ u = jnp.where(row_keep[None, None, :, None] > 0, u, -jnp.inf)+ w = jax.nn.softmax(u.reshape(b, q, l1 * k), axis=-1)+ return jnp.einsum("bqm,bmd->bqd", w, r.reshape(b, l1 * k, d)+ ).reshape(b, q * d).astype(dtype)-class ContrastiveHead(nn.Module):- d_model: int- out_dim: int- dtype: jnp.dtype = jnp.bfloat16- temp_init: float = 0.07+def head_config(config, key, probes, queries):+ name = key.removesuffix("_head")+ return replace(config, **{f"{name}_probes": probes, f"{name}_queries": queries})+++class HeadProjection(nn.Module):+ features: int+ use_bias: bool = True+ dtype: jnp.dtype = None++ @nn.compact+ def __call__(self, x, quant=False):+ kernel = head_weight(+ self.param("kernel", default_init(), (x.shape[-1], self.features)),+ quant, reduce_second_last=True)+ if self.dtype is not None:+ x, kernel = x.astype(self.dtype), kernel.astype(self.dtype)+ y = x @ kernel+ if self.use_bias:+ bias = self.param("bias", jinit.zeros, (self.features,))+ y = y + (bias.astype(self.dtype) if self.dtype is not None else bias)+ return y+++class ProbeHead(nn.Module):+ config: TransformerConfig+ dtype: jnp.dtype = None+ key = ""+ code = 0+ out_dim = 0++ @classmethod+ def export(cls, head):+ kernel = np.asarray(head["proj"]["kernel"]).T+ bias = (np.asarray(head["proj"]["bias"]) if "bias" in head["proj"]+ else np.zeros(kernel.shape[0], np.float16))+ return [(f"{cls.key}.probes", np.asarray(head["probes"])),+ (f"{cls.key}.gain", np.asarray(head["gain"])),+ (f"{cls.key}.query", np.asarray(head["query"])),+ (f"{cls.key}.row_bias", np.asarray(head["row_bias"])),+ (f"{cls.key}.proj", kernel),+ (f"{cls.key}.bias", bias)]++ def pooled(self, cells, keep, row_keep, quant=False):+ name = self.key.removesuffix("_head")+ l1, d = cells.shape[2], cells.shape[3]+ k, q = getattr(self.config, f"{name}_probes"), getattr(self.config, f"{name}_queries")+ return probe_pool(cells, keep,+ head_weight(self.param("probes", default_init(), (l1, k, d)), quant),+ self.param("gain", jinit.ones, (l1, k)),+ head_weight(self.param("query", default_init(), (q, d)), quant),+ self.param("row_bias", jinit.zeros, (q, l1, k)),+ row_keep, self.dtype or self.config.jax_dtype)- PROBES = 4+ @nn.compact+ def __call__(self, cells, keep=None, row_keep=None, quant=False):+ logits = HeadProjection(self.out_dim, dtype=self.dtype or self.config.jax_dtype,+ use_bias=True, name="proj")(+ self.pooled(cells, keep, row_keep, quant), quant)+ return logits.astype(jnp.float32)+++class ConfidenceHead(ProbeHead):+ key, code, out_dim = "confidence_head", 2, 1+++class RouterHead(ProbeHead):+ key, code, out_dim = "router_head", 3, 3+ calibration_init = (0.90, 0.00, 0.60)++ @classmethod+ def export(cls, head):+ calibration = np.asarray(head.get("calibration", cls.calibration_init), np.float16)+ return super().export(head) + [(f"{cls.key}.calibration", calibration)]+++class EmbeddingHead(ProbeHead):+ key, code = "embedding_head", 1+ temp_init: float = 10.0+ bias_init: float = -10.0@nn.compact- def __call__(self, cells, keep=None):- pooled = probe_pool(cells, self.param("probes", default_init(),- (self.PROBES, cells.shape[-1])),- keep, self.dtype)- p = nn.Dense(self.out_dim, dtype=self.dtype, use_bias=False,- kernel_init=default_init(), name="proj")(pooled)+ def __call__(self, cells, keep=None, row_keep=None, quant=False):+ p = HeadProjection(self.config.embedding_dim, dtype=self.dtype or self.config.jax_dtype,+ use_bias=False, name="proj")(+ self.pooled(cells, keep, row_keep, quant), quant)log_temp = self.param("log_temp", jinit.constant(math.log(self.temp_init)), ())+ bias = self.param("bias", jinit.constant(self.bias_init), ())denom = jnp.sqrt(jnp.sum(p.astype(jnp.float32) ** 2, axis=-1, keepdims=True) + 1e-12)- return (p / denom.astype(p.dtype)), log_temp+ return p / denom.astype(p.dtype), log_temp, bias-class ConfidenceHead(nn.Module):- dtype: jnp.dtype = jnp.bfloat16+HEADS = (EmbeddingHead, ConfidenceHead, RouterHead)+HEAD_KEYS = tuple(head.key for head in HEADS)- PROBES = 8- @nn.compact- def __call__(self, cells, keep=None):- pooled = probe_pool(cells, self.param("probes", default_init(),- (self.PROBES, cells.shape[-1])),- keep, self.dtype)- logit = nn.Dense(1, dtype=self.dtype, use_bias=True,- kernel_init=default_init(), name="proj")(pooled)- return logit[..., 0].astype(jnp.float32)+def init_head(params, model, head, seed):+ cfg = model.config+ f32 = lambda tree: jax.tree.map(lambda x: jnp.asarray(x, jnp.float32), tree)+ dummy = jnp.zeros((1, 8, cfg.num_layers + 1, cfg.d_model))+ init = head.init({"params": jax.random.PRNGKey(seed)}, dummy,+ keep=jnp.ones((1, 8)))["params"]+ return f32({k: v for k, v in params.items() if k != head.key}), {head.key: f32(init)}class SimpleAttentionNetwork(nn.Module):@@ -481,11 +1010,25 @@ class SimpleAttentionNetwork(nn.Module):def setup(self):cfg = self.configself.embedding = nn.Embed(cfg.vocab_size, cfg.d_model, embedding_init=jinit.normal(stddev=0.02))+ if cfg.image_sem_codes:+ if (cfg.image_sem_dim <= 0 or cfg.image_tex_books <= 0+ or cfg.image_tex_codes <= 0 or cfg.image_tex_stages <= 0+ or cfg.image_tex_dim <= 0):+ raise ValueError("incomplete composite-image embedding geometry")+ self.image_sem_embedding = nn.Embed(+ cfg.image_sem_codes, cfg.image_sem_dim,+ embedding_init=jinit.normal(stddev=0.02))+ self.image_texture_embedding = nn.Embed(+ cfg.image_tex_books * cfg.image_tex_codes * cfg.image_tex_stages,+ cfg.image_tex_dim, embedding_init=jinit.normal(stddev=0.02))+ self.image_projection = nn.Dense(+ cfg.d_model, use_bias=False, kernel_init=default_init())+ self.image_input_norm = ZCRMSNorm(dtype=cfg.jax_dtype)self.embed_scale = math.sqrt(cfg.d_model)self.stack = Stack(cfg)- self.contrastive_head = ContrastiveHead(- cfg.d_model, cfg.contrastive_dim, cfg.jax_dtype)- self.confidence_head = ConfidenceHead(cfg.jax_dtype)+ self.embedding_head = EmbeddingHead(cfg)+ self.confidence_head = ConfidenceHead(cfg)+ self.router_head = RouterHead(cfg)assert all(l < cfg.num_layers for l in cfg.engram_layers)orders, heads, sub_dim = engram_geometry(cfg)self.engrams = [@@ -493,52 +1036,93 @@ class SimpleAttentionNetwork(nn.Module):sub_dim, cfg.num_layers, max(orders), cfg.jax_dtype)for _ in cfg.engram_layers]- self.mtp_combine = nn.Dense(cfg.d_model, dtype=cfg.jax_dtype, use_bias=False,- kernel_init=default_init())- self.mtp_block = Block(cfg.num_heads, cfg.num_kv_heads, cfg.d_model,- cfg.num_layers, cfg.jax_dtype, cfg.flash,- attn_dim=cfg.attn_dim)- self.mtp_emb_norm = ZCRMSNorm(dtype=cfg.jax_dtype)- self.mtp_final_norm = ZCRMSNorm(dtype=cfg.jax_dtype)def _rope(self, seq_len):cfg = self.config- head_dim = (cfg.attn_dim or cfg.d_model) // cfg.num_heads- return precompute_rope_freqs(head_dim, seq_len, cfg.rope_theta)+ qk_hd, _ = head_dims(cfg)+ return precompute_rope_freqs(qk_hd, seq_len, cfg.rope_theta)def _engram_kv(self, tokens, mask, quant):if not self.engrams:return Noneorders, heads, _ = engram_geometry(self.config)- indices = engram_indices(tokens, orders, heads, self.config.engram_slots)+ indices = engram_indices(tokens, orders, heads, self.config.engram_slots,+ getattr(self.config, "engram_seed_heads", 0))ngram_ok = jnp.stack([_mask_diag(mask, o - 1) for o in orders for _ in range(heads)],axis=-1)tap_ok = jnp.stack([_mask_diag(mask, j * max(orders)) for j in range(ENGRAM_CONV_TAPS)])pairs = [e(indices, ngram_ok, tap_ok, quant=quant) for e in self.engrams]return jnp.stack([k for k, _ in pairs]), jnp.stack([v for _, v in pairs])- def __call__(self, tokens, mask=None, quant=False, return_mtp=False):+ def _input_embeddings(self, tokens, image_texture=None):+ cfg = self.config+ x = self.embedding(tokens) * self.embed_scale+ if image_texture is None:+ if not (self.is_initializing() and cfg.image_sem_codes):+ return x+ image_texture = jnp.full(+ tokens.shape + (cfg.image_tex_books * cfg.image_tex_stages,),+ 255, dtype=jnp.uint8)+ if not cfg.image_sem_codes:+ raise ValueError("image components were provided to a text-only model")+ expected = cfg.image_tex_books * cfg.image_tex_stages+ if image_texture.shape != tokens.shape + (expected,):+ raise ValueError(+ f"image components must have shape tokens+({expected},)")+ active = image_texture[..., 0] != 255+ texture = jnp.where(active[..., None], image_texture, 0).astype(jnp.int32)+ semantic = jnp.clip(+ tokens.astype(jnp.int32) - cfg.image_code_offset,+ 0, cfg.image_sem_codes - 1)+ sem = self.image_sem_embedding(semantic)+ offsets = (jnp.arange(expected, dtype=jnp.int32)+ * cfg.image_tex_codes)+ tex = self.image_texture_embedding(texture + offsets)+ tex = tex.reshape(+ *tokens.shape, cfg.image_tex_stages, cfg.image_tex_books,+ cfg.image_tex_dim).sum(axis=-3)+ composite = jnp.concatenate(+ [sem, tex.reshape(*tokens.shape,+ cfg.image_tex_books * cfg.image_tex_dim)],+ axis=-1)+ image_x = self.image_projection(composite)+ image_x = (self.image_input_norm(image_x)+ * (0.02 * self.embed_scale)).astype(x.dtype)+ return jnp.where(active[..., None], image_x, x)++ def __call__(self, tokens, mask=None, quant=False,+ image_texture=None, exit_depth=None, exit_only=False,+ subnetwork_only=False):if mask is None:mask = make_causal_mask(tokens.shape[1])- x = self.embedding(tokens) * self.embed_scale+ x = self._input_embeddings(tokens, image_texture)rope = self._rope(tokens.shape[1])engram_kv = self._engram_kv(tokens, mask, quant)- x, _ = self.stack(x, mask=mask, rope=rope, engram_kv=engram_kv, quant=quant)- logits = _aq(x, quant).astype(jnp.float32) @ self.embedding.embedding.T- if not (return_mtp or self.is_initializing()):- return logits-- nxt = jnp.pad(tokens[:, 1:], ((0, 0), (0, 1)))- e2 = self.mtp_emb_norm(self.embedding(nxt) * self.embed_scale)- m = self.mtp_combine(_aq(jnp.concatenate([x, e2], axis=-1), quant))- m = self.mtp_block(m, mask=mask, rope=rope, quant=quant)- m = self.mtp_final_norm(m)- mtp_logits = _aq(m, quant).astype(jnp.float32) @ self.embedding.embedding.T- if not return_mtp:- return logits- return logits, mtp_logits-- def hidden_cells(self, tokens, quant=False, window=0, sink=None):+ stack_out = self.stack(+ x, mask=mask, rope=rope, engram_kv=engram_kv, quant=quant,+ exit_depth=exit_depth, subnetwork_only=subnetwork_only)+ if subnetwork_only:+ x, _ = stack_out+ exit_x = None+ elif exit_depth is None:+ x, _ = stack_out+ exit_x = None+ else:+ x, _, exit_x = stack_out+ head = self.embedding.embedding+ if self.config.out_vocab:+ head = head[: self.config.out_vocab]+ if subnetwork_only:+ return _aq(x, quant).astype(jnp.float32) @ head.T+ if exit_x is not None:+ exit_logits = _aq(exit_x, quant).astype(jnp.float32) @ head.T+ if exit_only:+ return exit_logits+ logits = _aq(x, quant).astype(jnp.float32) @ head.T+ return (logits, exit_logits) if exit_x is not None else logits++ def hidden_cells(self, tokens, quant=False, window=0, sink=None,+ image_texture=None, exit_depth=None):cfg = self.configmask = (make_causal_mask(tokens.shape[1])& make_padding_mask(tokens, cfg.pad_token_id))@@ -547,41 +1131,45 @@ class SimpleAttentionNetwork(nn.Module):recent = ((pos[:, None] - pos[None, :]) < window)[None, None, :, :]keep = recent if sink is None else (recent | sink[:, None, None, :])mask = mask & keep- x0 = self.embedding(tokens) * self.embed_scale+ x0 = self._input_embeddings(tokens, image_texture)rope = self._rope(tokens.shape[1])engram_kv = self._engram_kv(tokens, mask, quant)- _, hidden = self.stack(x0, mask=mask, rope=rope, engram_kv=engram_kv,- quant=quant, collect_hidden=True)+ hidden = self.stack(+ x0, mask=mask, rope=rope, engram_kv=engram_kv,+ quant=quant, collect_hidden=True, exit_depth=exit_depth)[1]return jnp.stack([x0, *hidden], axis=2)- def _encode_contrastive(self, tokens, quant=False, window=0, sink=None):+ def _head_cells(self, tokens, quant=False, window=0, sink=None,+ exit_depth=None):cells = jax.lax.stop_gradient(- self.hidden_cells(tokens, quant=quant, window=window, sink=sink))+ self.hidden_cells(tokens, quant=quant, window=window, sink=sink,+ exit_depth=exit_depth))keep = (tokens != self.config.pad_token_id).astype(jnp.float32)- return self.contrastive_head(cells, keep=keep)+ return cells, keep, ladder_row_keep(self.config, exit_depth)- def encode_contrastive(self, tokens, quant=False, window=0, sink=None):- return self._encode_contrastive(tokens, quant=quant, window=window,- sink=sink)[0]+ def forward_embedding(self, tokens, quant=False, window=0, sink=None,+ exit_depth=None):+ return self.embedding_head(+ *self._head_cells(tokens, quant, window, sink, exit_depth), quant=quant)[0]- def forward_contrastive(self, query_tokens, tool_tokens, quant=False):- q_emb, log_temp = self._encode_contrastive(query_tokens, quant=quant)- t_emb, _ = self._encode_contrastive(tool_tokens, quant=quant)- return q_emb, t_emb, log_temp+ def forward_confidence(self, tokens, quant=False, window=0, sink=None,+ exit_depth=None):+ return self.confidence_head(+ *self._head_cells(tokens, quant, window, sink, exit_depth), quant=quant)[..., 0]- def forward_confidence(self, tokens, quant=False, window=0, sink=None):- cells = jax.lax.stop_gradient(- self.hidden_cells(tokens, quant=quant, window=window, sink=sink))- keep = (tokens != self.config.pad_token_id).astype(jnp.float32)- return self.confidence_head(cells, keep=keep)+ def forward_router(self, tokens, quant=False, window=0, sink=None,+ exit_depth=None):+ return self.router_head(+ *self._head_cells(tokens, quant, window, sink, exit_depth), quant=quant)- def hidden_states(self, tokens, mask=None):+ def hidden_states(self, tokens, mask=None, image_texture=None):if mask is None:mask = make_causal_mask(tokens.shape[1])- x = self.embedding(tokens) * self.embed_scale+ x = self._input_embeddings(tokens, image_texture)rope = self._rope(tokens.shape[1])engram_kv = self._engram_kv(tokens, mask, False)- _, hidden = self.stack(x, mask=mask, rope=rope, engram_kv=engram_kv, collect_hidden=True)+ _, hidden = self.stack(x, mask=mask, rope=rope, engram_kv=engram_kv,+ collect_hidden=True)return hidden.astype(jnp.float32)@@ -601,12 +1189,25 @@ KV_WINDOW_MIN = 160def kv_budget_window(config):- head_dim = (getattr(config, "attn_dim", 0) or config.d_model) // config.num_heads- kv = config.num_kv_heads * head_dim+ qk_hd, v_hd = head_dims(config)+ kd = config.num_kv_heads * qk_hd+ vd = config.num_kv_heads * v_hdd, L = config.d_model, config.num_layerssites = len(tuple(getattr(config, "engram_layers", (2, 15))))- per_pos = (L * (2 * kv + 2 * (kv // KV_GROUP) * 4)- + sites * (d + (d // KV_GROUP) * 4))+ per_layer = kd + vd + (kd // KV_GROUP + vd // KV_GROUP) * 4+ per_site = d + (d // KV_GROUP) * 4+ sw = getattr(config, "sliding_window", 0)+ if sw:+ n_global = len(tuple(getattr(config, "global_layers", ())))+ local = min(sw, config.max_seq_len)+ fixed = ((L - n_global) * per_layer + sites * per_site) * local+ if n_global == 0:+ return config.max_seq_len if fixed <= KV_BUDGET_BYTES else \+ max(KV_WINDOW_MIN, local)+ ctx = ((KV_BUDGET_BYTES - fixed) // (n_global * per_layer)+ // KV_GROUP * KV_GROUP)+ return max(KV_WINDOW_MIN, min(ctx, config.max_seq_len))+ per_pos = L * per_layer + sites * per_sitewindow = (KV_BUDGET_BYTES // per_pos) // KV_GROUP * KV_GROUPreturn max(KV_WINDOW_MIN, min(window, config.max_seq_len))@@ -621,9 +1222,11 @@ def make_causal_packing_mask(seg_ids, prefix=None, window=0):causal = jnp.tril(jnp.ones((T, T), dtype=jnp.bool_))block = (seg_ids[:, :, None] == seg_ids[:, None, :]) & (seg_ids[:, :, None] > 0)mask = block & causal[None, :, :]- if window:+ static_zero = isinstance(window, (int, np.integer)) and int(window) == 0+ if window is not None and not static_zero:pos = jnp.arange(T)- recent = (pos[:, None] - pos[None, :]) < window+ limit = jnp.where(jnp.asarray(window) > 0, window, T + 1)+ recent = (pos[:, None] - pos[None, :]) < limitsink = (jnp.zeros_like(seg_ids, dtype=jnp.bool_) if prefix is Noneelse prefix > 0)mask = mask & (recent[None, :, :] | sink[:, None, :])diff --git a/needle/model/checkpoints.py b/needle/model/checkpoints.pyindex 141ff61..e8f8b1b 100644--- a/needle/model/checkpoints.py+++ b/needle/model/checkpoints.py@@ -92,7 +92,7 @@ def write_checkpoint(path, checkpoint):pickle.dump(checkpoint, handle)-_ADAPTER_FIELDS = ("scale", "base", "rank", "qat_bits", "qat_bits_map", "seed")+_ADAPTER_FIELDS = ("scale", "base", "rank", "qat_bits", "qat_bits_map", "seed", "layers")def write_adapter(path, adapter):diff --git a/needle/model/decode.py b/needle/model/decode.pydeleted file mode 100644index 3dbacd2..0000000--- a/needle/model/decode.py+++ /dev/null@@ -1,405 +0,0 @@-import math-from collections import namedtuple--import jax-import jax.numpy as jnp--from .architecture import (precompute_rope_freqs, apply_rope, _walsh_matrix,- engram_geometry, engram_indices, _rms_unit, _shift_right,- _sinkhorn, ENGRAM_CONV_TAPS)---def _doc_prefix_len(ids):- return 0--DecodeCfg = namedtuple("DecodeCfg", [- "d_model", "num_heads", "num_kv_heads", "num_layers", "attn_dim", "mhc_lanes",- "engram_layers", "engram_orders", "engram_heads", "engram_slots",- "kv_window",-])---def _attn_width(config):- return getattr(config, "attn_dim", 0) or config.d_model---def decode_cfg(config, kv_window=0):- orders, heads, _ = engram_geometry(config)- return DecodeCfg(config.d_model, config.num_heads, config.num_kv_heads,- config.num_layers, _attn_width(config), config.mhc_lanes,- tuple(config.engram_layers), tuple(orders), heads,- config.engram_slots, int(kv_window))---def init_kv_cache(config, batch, max_len):- head_dim = _attn_width(config) // config.num_heads- shape = (config.num_layers, batch, config.num_kv_heads, max_len, head_dim)- z = jnp.zeros(shape, jnp.float32)- return z, z---def _zcrms(x, scale, eps=1e-6):- rms = jnp.sqrt(jnp.mean(x ** 2, axis=-1, keepdims=True) + eps)- return (1.0 + scale) * x / rms---def _layer(params, i):- b = params["stack"]["layers"]["block"]- sa = b["self_attn"]- ha = b["hadamard_mlp"]- lp = {- "pre": b["ZCRMSNorm_0"]["scale"][i],- "q": sa["q_proj"]["kernel"][i], "k": sa["k_proj"]["kernel"][i],- "v": sa["v_proj"]["kernel"][i], "o": sa["out_proj"]["kernel"][i],- "attn_gate_proj": sa["gate_proj"]["kernel"][i],- "qn": sa["q_norm"]["scale"][i], "kn": sa["k_norm"]["scale"][i],- "post": b["post_attn_norm"]["scale"][i], "attn_gate": b["attn_gate"][i],- "pre_hada": b["pre_hada_norm"]["scale"][i],- "hd1": ha["d1"][i], "hd2": ha["d2"][i], "hd3": ha["d3"][i],- }- return lp---def _attn_cached(x, lp, k_cache_l, v_cache_l, pos, cos_s, sin_s, cfg, key_valid,- flash=False, sink=None):- B, S, _ = x.shape- H, KV = cfg.num_heads, cfg.num_kv_heads- hd = cfg.attn_dim // H-- q = (x @ lp["q"]).reshape(B, S, H, hd).transpose(0, 2, 1, 3)- k = (x @ lp["k"]).reshape(B, S, KV, hd).transpose(0, 2, 1, 3)- v = (x @ lp["v"]).reshape(B, S, KV, hd).transpose(0, 2, 1, 3)- q = _zcrms(q, lp["qn"]); k = _zcrms(k, lp["kn"])- q = apply_rope(q, cos_s, sin_s); k = apply_rope(k, cos_s, sin_s)-- k_cache_l = jax.lax.dynamic_update_slice(k_cache_l, k, (0, 0, pos, 0))- v_cache_l = jax.lax.dynamic_update_slice(v_cache_l, v, (0, 0, pos, 0))- max_len = k_cache_l.shape[2]-- if flash and S > 1:- causal = jnp.tril(jnp.ones((S, S), bool))- if cfg.kv_window:- rows = jnp.arange(S)- causal = causal & ((rows[:, None] - rows[None, :]) < cfg.kv_window)- if sink is not None:- causal = causal | (jnp.tril(jnp.ones((S, S), bool))- & sink[:, :S][:, None, :])- if key_valid is None:- m = causal[None, None, :, :] if causal.ndim == 2 else causal[:, None, :, :]- else:- c = causal[None] if causal.ndim == 2 else causal- m = (c & key_valid[:, :S][:, None, :]) | jnp.eye(S, dtype=bool)[None]- m = m[:, None, :, :]- impl = ("cudnn" if jax.default_backend() == "gpu"- and q.dtype in (jnp.bfloat16, jnp.float16) else None)- out = jax.nn.dot_product_attention(- q.transpose(0, 2, 1, 3), k.transpose(0, 2, 1, 3), v.transpose(0, 2, 1, 3),- mask=m, implementation=impl)- out = out.reshape(B, S, cfg.attn_dim)- else:- reps = H // KV- scale = math.sqrt(hd)- qg = q.reshape(B, KV, reps, S, hd)- aw = jnp.einsum("bgrsd,bgld->bgrsl", qg, k_cache_l) / scale- qpos = pos + jnp.arange(S)- kpos = jnp.arange(max_len)- causal = kpos[None, :] <= qpos[:, None]- if cfg.kv_window:- recent = (qpos[:, None] - kpos[None, :]) < cfg.kv_window- keep = recent[None] if sink is None else (recent[None] | sink[:, None, :])- causal = causal[None] & keep- else:- causal = causal[None]- if key_valid is None:- mask = causal[:, None, None, :, :]- else:- mask = (causal & key_valid[:, None, :])[:, None, None, :, :]- aw = jnp.where(mask, aw, jnp.finfo(aw.dtype).min)- aw = jax.nn.softmax(aw, axis=-1)- out = jnp.einsum("bgrsl,bgld->bgrsd", aw, v_cache_l)- out = out.reshape(B, H, S, hd).transpose(0, 2, 1, 3).reshape(B, S, cfg.attn_dim)-- out = out * jax.nn.sigmoid(x @ lp["attn_gate_proj"])- return out @ lp["o"], k_cache_l, v_cache_l---def _hadamard(z, d1, d2, d3, d_model):- n = 1 << (d_model - 1).bit_length()- H = _walsh_matrix(n).astype(z.dtype)- pad = n - d_model- if pad:- z = jnp.pad(z, ((0, 0), (0, 0), (0, pad)))- z = (d1 * z) @ H- z = jax.nn.silu(d2 * z) @ H- return (d3 * z)[..., :d_model]---def _block_cached(x, lp, k_cache_l, v_cache_l, pos, cos_s, sin_s, cfg, key_valid,- flash=False, sink=None):- skip = x- h = _zcrms(x, lp["pre"])- attn, k_cache_l, v_cache_l = _attn_cached(h, lp, k_cache_l, v_cache_l, pos,- cos_s, sin_s, cfg, key_valid, flash, sink)- attn = _zcrms(attn, lp["post"])- x = skip + jax.nn.sigmoid(lp["attn_gate"]) * attn-- skip = x- h = _zcrms(x, lp["pre_hada"])- x = skip + _hadamard(h, lp["hd1"], lp["hd2"], lp["hd3"], cfg.d_model)- return x, k_cache_l, v_cache_l---def _engram_window(cfg):- return ENGRAM_CONV_TAPS * max(cfg.engram_orders) if cfg.engram_layers else 0---def _engram_kv(params, cfg, hist, hist_valid, pos, S):- W = _engram_window(cfg)- B = hist.shape[0]- padded = jnp.pad(hist, ((0, 0), (W, 0)))- win = jax.lax.dynamic_slice(padded, (0, pos), (B, W + S))- pv = jnp.pad(hist_valid, ((0, 0), (W, 0)))- win_valid = jax.lax.dynamic_slice(pv, (0, pos), (B, W + S)).astype(jnp.float32)-- orders, heads = cfg.engram_orders, cfg.engram_heads- idx = engram_indices(win, orders, heads, cfg.engram_slots)- ngram_ok = jnp.stack([_shift_right(win_valid, o - 1)- for o in orders for _ in range(heads)], axis=-1)- tap_ok = jnp.stack([_shift_right(win_valid, j * max(orders))- for j in range(ENGRAM_CONV_TAPS)])-- ks, vs = [], []- for site in range(len(cfg.engram_layers)):- ep = params[f"engrams_{site}"]- tables = ep["embedding"]- fetched = tables[jnp.arange(tables.shape[0]), idx] * ngram_ok[..., None]- e = fetched.reshape(B, W + S, -1)- k = e @ ep["key_proj"]["kernel"]- v = e @ ep["value_proj"]["kernel"]- taps = ep["taps"]- v = sum(taps[j] * _shift_right(v, j * max(orders)) * tap_ok[j][..., None]- for j in range(ENGRAM_CONV_TAPS))- ks.append(k[:, W:]); vs.append(v[:, W:])- return jnp.stack(ks), jnp.stack(vs)---def _mhc(params, cfg):- st = params["stack"]- n, L = cfg.mhc_lanes, cfg.num_layers- lane = jnp.eye(n, dtype=jnp.float32)[jnp.arange(L) % n]- return {"phi_pre": st["mhc_phi_pre"], "phi_post": st["mhc_phi_post"],- "phi_res": st["mhc_phi_res"], "b_pre": st["mhc_b_pre"],- "b_post": st["mhc_b_post"], "b_res": st["mhc_b_res"],- "a_pre": st["mhc_a_pre"], "a_post": st["mhc_a_post"],- "a_res": st["mhc_a_res"], "pre_off": 8 * lane - 4,- "post_off": -4 * (1 - lane)}---def _forward_cached(params, cfg, tokens, k_cache, v_cache, pos, cos, sin, key_valid=None,- flash=False, hist=None, hist_valid=None, sink=None):- emb = params["embedding"]["embedding"].astype(jnp.float32)- x = emb[tokens] * math.sqrt(cfg.d_model)- B, S = tokens.shape- cos_s = jax.lax.dynamic_slice_in_dim(cos, pos, S, axis=0)- sin_s = jax.lax.dynamic_slice_in_dim(sin, pos, S, axis=0)-- ekv = None- if cfg.engram_layers:- h = tokens if hist is None else hist- hv = jnp.ones(h.shape, bool) if hist_valid is None else hist_valid- ekv = _engram_kv(params, cfg, h, hv,- jnp.asarray(0, jnp.int32) if hist is None else pos, S)-- n, C = cfg.mhc_lanes, cfg.d_model- hc = _mhc(params, cfg)- x = jnp.broadcast_to(x[:, :, None, :], (B, S, n, C))-- new_k, new_v = [], []- for i in range(cfg.num_layers):- nx = _rms_unit(x.reshape(B, S, n * C))- hpre = jax.nn.sigmoid(hc["a_pre"][i] * (nx @ hc["phi_pre"][i])- + hc["b_pre"][i] + hc["pre_off"][i])- u = jnp.einsum("btn,btnc->btc", hpre, x)- bx = u- if ekv is not None and i in cfg.engram_layers:- site = cfg.engram_layers.index(i)- ek, ev = ekv- alpha = jax.nn.sigmoid(- jnp.einsum("btd,btd->bt", _rms_unit(u), _rms_unit(ek[site]))- / math.sqrt(C))- bx = u + alpha[..., None] * ev[site]- y, kc, vc = _block_cached(bx, _layer(params, i), k_cache[i], v_cache[i],- pos, cos_s, sin_s, cfg, key_valid, flash, sink)- y = y - u- hpost = 2 * jax.nn.sigmoid(hc["a_post"][i] * (nx @ hc["phi_post"][i])- + hc["b_post"][i] + hc["post_off"][i])- res = nx @ hc["phi_res"][i]- hres = _sinkhorn(hc["a_res"][i] * res.reshape(B, S, n, n) + hc["b_res"][i])- x = (jnp.einsum("btij,btjc->btic", hres, x)- + hpost[..., None] * y[:, :, None, :])- new_k.append(kc); new_v.append(vc)- x = jnp.mean(x, axis=2)- x = _zcrms(x, params["stack"]["final_norm"]["scale"])- logits = x @ emb.T- return logits, jnp.stack(new_k), jnp.stack(new_v)---forward_cached = jax.jit(_forward_cached, static_argnums=(1, 9))---def _f32(params):- return jax.tree.map(lambda a: jnp.asarray(a, jnp.float32), params)---def generate_cached(config, params, tokenizer, prompt, max_new_tokens=96, max_len=None,- kv_window=0):- from .tokenizer import BOS_ID, EOS_ID- params = _f32(params)- prompt_ids = ([BOS_ID] + tokenizer.encode(prompt))[:config.max_seq_len - 1]- max_len = max_len or min(config.max_seq_len, len(prompt_ids) + max_new_tokens)- head_dim = _attn_width(config) // config.num_heads- cos, sin = precompute_rope_freqs(head_dim, max_len, config.rope_theta)-- dcfg = decode_cfg(config, kv_window=kv_window)- kc, vc = init_kv_cache(config, 1, max_len)- hist = jnp.zeros((1, max_len), jnp.int32).at[0, :len(prompt_ids)].set(- jnp.asarray(prompt_ids, jnp.int32))- valid = jnp.ones((1, max_len), bool)- sink = None- if kv_window:- import numpy as _np- sk = _np.zeros((1, max_len), bool)- sk[0, :_doc_prefix_len(prompt_ids)] = True- sink = jnp.asarray(sk)- toks = jnp.asarray([prompt_ids], jnp.int32)- logits, kc, vc = forward_cached(params, dcfg, toks, kc, vc,- jnp.asarray(0, jnp.int32), cos, sin, None, False,- hist, valid, sink)- nxt = int(jnp.argmax(logits[0, -1]))-- out, pos = [], len(prompt_ids)- for _ in range(max_new_tokens):- if nxt == EOS_ID or pos >= max_len:- break- out.append(nxt)- hist = hist.at[0, pos].set(nxt)- logits, kc, vc = forward_cached(params, dcfg, jnp.asarray([[nxt]], jnp.int32),- kc, vc, jnp.asarray(pos, jnp.int32), cos, sin,- None, False, hist, valid, sink)- nxt = int(jnp.argmax(logits[0, -1]))- pos += 1- return tokenizer.decode(out)---def _shard(x, nd):- return x.reshape(nd, x.shape[0] // nd, *x.shape[1:])---def _sample(last, rng, temperature):- logp = jax.nn.log_softmax(last, axis=-1)- nxt = jnp.where(temperature > 0,- jax.random.categorical(rng, last / jnp.maximum(temperature, 1e-6), axis=-1),- jnp.argmax(last, axis=-1)).astype(jnp.int32)- tok_logp = jnp.take_along_axis(logp, nxt[:, None], axis=-1)[:, 0]- return nxt, tok_logp---def _make_rollout(cfg, max_len, scan_len):- hd = cfg.attn_dim // cfg.num_heads-- def rollout(params, buf, valid, sink, cos, sin, rng, temperature):- B, max_p = buf.shape- z = jnp.zeros((cfg.num_layers, B, cfg.num_kv_heads, max_len, hd), jnp.float32)- hist = jnp.zeros((B, max_len), jnp.int32).at[:, :max_p].set(buf)- logits, kc, vc = _forward_cached(params, cfg, buf, z, z,- jnp.asarray(0, jnp.int32), cos, sin, valid,- flash=True, hist=hist, hist_valid=valid,- sink=sink)- rng, sk = jax.random.split(rng)- t0, lp0 = _sample(logits[:, -1], sk, temperature)-- def body(carry, _):- kc, vc, last, pos, rng, hist = carry- hist = jax.lax.dynamic_update_slice(hist, last[:, None], (0, pos))- logits, kc, vc = _forward_cached(params, cfg, last[:, None], kc, vc,- pos, cos, sin, valid,- hist=hist, hist_valid=valid, sink=sink)- rng, sk = jax.random.split(rng)- tok, lp = _sample(logits[:, -1], sk, temperature)- return (kc, vc, tok, pos + 1, rng, hist), (tok, lp)-- init = (kc, vc, t0, jnp.asarray(max_p, jnp.int32), rng, hist)- _, (toks, lps) = jax.lax.scan(body, init, None, length=scan_len)- toks = jnp.concatenate([t0[None], toks], axis=0)- lps = jnp.concatenate([lp0[None], lps], axis=0)- return toks.T, lps.T- return rollout---_rollout_cache = {}---def _rollout_fn(cfg, max_len, scan_len):- key = (cfg, max_len, scan_len)- if key not in _rollout_cache:- _rollout_cache[key] = jax.pmap(_make_rollout(cfg, max_len, scan_len),- in_axes=(0, 0, 0, 0, None, None, 0, None))- return _rollout_cache[key]---def batched_generate(config, params, tokenizer, prompts, max_new_tokens=96,- temperature=0.0, seed=0, pad_to=None, kv_window=0):- from .tokenizer import BOS_ID, EOS_ID, PAD_ID- import numpy as _np-- params = _f32(params)- dcfg = decode_cfg(config, kv_window=kv_window)- cap = max(1, config.max_seq_len - max_new_tokens)- enc = [([BOS_ID] + tokenizer.encode(p))[:cap] for p in prompts]- n_real = len(enc)-- nd = jax.local_device_count()- B = ((n_real + nd - 1) // nd) * nd- while len(enc) < B:- enc.append([BOS_ID])- max_p = min(cap, pad_to) if pad_to else max(len(e) for e in enc)- max_len = min(config.max_seq_len, max_p + max_new_tokens)- scan_len = max(max_len - max_p - 1, 0)- if scan_len == 0:- raise ValueError(- f"longest prompt is {max_p} tokens, leaving no room for "- f"{max_new_tokens} new ones inside max_seq_len {config.max_seq_len} - "- f"the whole batch would generate a single token. Drop prompts longer "- f"than max_seq_len - max_new_tokens before calling.")-- buf = _np.full((B, max_p), PAD_ID, _np.int32)- valid = _np.zeros((B, max_len), bool)- sink = _np.zeros((B, max_len), bool)- for b, e in enumerate(enc):- start = max_p - len(e)- buf[b, start:] = e- valid[b, start:] = True- if kv_window:- sink[b, start:start + _doc_prefix_len(e)] = True- valid[:, max_p:] = True-- head_dim = _attn_width(config) // config.num_heads- cos, sin = precompute_rope_freqs(head_dim, max_len, config.rope_theta)-- pr = jax.tree.map(lambda x: jnp.broadcast_to(x, (nd,) + x.shape), params)- rollout = _rollout_fn(dcfg, max_len, scan_len)- rngs = jax.random.split(jax.random.PRNGKey(seed), nd)- toks, lps = rollout(pr, _shard(jnp.asarray(buf), nd), _shard(jnp.asarray(valid), nd),- _shard(jnp.asarray(sink), nd), cos, sin, rngs,- jnp.asarray(temperature, jnp.float32))- toks = _np.asarray(toks).reshape(B, -1)- lps = _np.asarray(lps).reshape(B, -1)-- keep = _np.cumprod(toks != EOS_ID, axis=1).astype(bool)- texts, out_ids, out_lps = [], [], []- for b in range(n_real):- ids = toks[b][keep[b]].tolist()- out_ids.append(ids)- out_lps.append(lps[b][keep[b]].tolist())- texts.append(tokenizer.decode(ids))- return texts, out_ids, out_lpsdiff --git a/needle/model/export.py b/needle/model/export.pyindex 4907f23..d647878 100644--- a/needle/model/export.py+++ b/needle/model/export.py@@ -2,8 +2,8 @@weights the C++ / single-mega-kernel inference reads (mmap or linked read-onlysection; no parsing).-Format is W4A8: the matmul weights (tied embedding + every attention projection)-are Cactus-Quants INT4; norms, Hadamard diagonals, and gates stay FP16. Weights+Format is CQ-WxA8: matmul weights use the per-tensor CQ width selected by the+deployment map; norms, Hadamard diagonals, and gates stay FP16. Weightsare PRE-TRANSPOSED to [out, in] so each output row is contiguous along thereduction axis (what a GEMV/GEMM stream wants, and the axis quant groups runalong). Tensors are LAYER-MAJOR: embedding, then all of layer 0's tensors, ...,@@ -16,14 +16,21 @@ The runtime reads the geometry from the header, so one binary loads and runsany configuration of the (single) architecture; tensors are positional in thefixed canon order below.-HEADER (30 u32-sized fields, rope_theta an f32, everything else u32):+HEADER (49 u32-sized fields, rope_theta an f32, everything else u32):tag, num_tensors, codebook_len, kv_window, kv_bits,- vocab, d_model, num_heads, num_kv_heads, num_layers, head_dim, max_seq_len,- hada_n, mhc_lanes, engram_slots, engram_sub_dim, num_engram_tables,- engram_conv_taps, engram_conv_dilation, num_engram_orders, engram_orders[4],- num_engram_sites, engram_sites[4], rope_theta;+ vocab, out_vocab, d_model, num_heads, num_kv_heads, num_layers,+ qk_head_dim, v_head_dim, max_seq_len, hada_n, mhc_lanes,+ sliding_window, global_mask_lo, global_mask_hi, qkv_conv_taps,+ engram_slots, engram_sub_dim, num_engram_tables, engram_conv_taps,+ engram_conv_dilation, engram_seed_heads, num_engram_orders,+ engram_orders[4], num_engram_sites, engram_sites[16], rope_theta.+ out_vocab is the tied text-slice head (0 = full vocab); rows past out_vocab+ are input-only code embeddings. sliding_window is the per-layer local width;+ global_mask bits (layer i = bit i, lo then hi u32) mark full-attention+ layers. qkv_conv_taps is the causal depthwise conv width on Q,K,V (0 = none);then codebook_len * f32 - the shared Lloyd-Max unit-sphere codebooks,cb2[4] | cb3[8] | cb4[16] concatenated, so mixed-precision blobs need one+header. Binary CQ1 uses its analytic {-c, +c} codebook and does not enlarge theheader. kv_window is the sliding-window width the model was trained with (0 =size it from the KV budget alone); kv_bits is the KV-cache width it waspost-trained for (8 = int8, else 2/3/4). Both ride in the blob because a model@@ -33,20 +40,36 @@ serves the wrong numerics.DIRECTORY (num_tensors records, {REC_SIZE} bytes each, no names):u8 dtype, u8 ndim, u16 _pad, u32 shape[4], u64 offset, u64 nbytes,u32 group_size, u32 bits. dtype: 1=FP16, 2=FP32, 3=CQ (bits gives the- width: 2, 3 or 4), 4=RAW.+ width: 1, 2, 3 or 4; 5 denotes ternary crumbs), 4=RAW.TENSOR ORDER (what the runtime indexes by position):- embedding; per layer [norm_in, q_proj, k_proj, v_proj, q_norm, k_norm,- gate_proj, out_proj, post_norm, attn_gate, pre_hada, d1, d2, d3]; the 9 mHC- blocks; per engram site [tables, key_proj, value_proj, taps]; final_norm;- the optional probe heads (see below); then the RAW tokenizer.--PROBE HEADS (all FP16, appended so pre-head blobs stay loadable). Let+ embedding; per layer [norm_in, q_proj, k_proj, v_proj,+ (q_taps, k_taps, v_taps when qkv_conv_taps > 0), q_norm, k_norm,+ gate_proj, out_proj, post_norm, attn_gate, pre_hada, d1, d2, b2, d3, d4,+ w1a, w1b, w2a, w2b, w3a, w3b, cond_v, cond_u]; the 9 mHC blocks; per engram+ site [tables, key_proj, value_proj, taps]; final_norm; optional composite+ image tensors [manifest, semantic table, texture table, projection,+ input_norm]; the optional probe heads (see below); then the RAW tokenizer.++The image manifest is FP32 with fields [magic, code_offset, positions,+semantic_codes, semantic_dim, texture_books, texture_codes, texture_stages,+texture_dim]. The vision encoder transports semantic IDs first, followed by+positions * texture_books * texture_stages texture IDs. The runtime consumes+that buffer as `positions` LM inputs rather than flattening every code into a+separate token.++PROBE HEADS (appended so pre-head blobs stay loadable). Letextra = num_tensors - (through final_norm) - (tokenizer). extra == 0 means noheads. Otherwise the tensor right after final_norm is a `heads.manifest` of- shape (H,), one code per head in canonical order (1 contrastive,- 2 confidence), followed by H fixed-stride triples [probes (P, d_model),- proj (out, P*d_model), bias (out,)], so H = (extra - 1) / 3.+ shape (H,), one code per head in canonical order (1 embedding,+ 2 confidence, 3 router), then each head's tensors with its own k and q:+ [probes CQ ((L+1)*k, d_model), gain FP16 (L+1, k), query CQ (q, d_model),+ row_bias FP16 (q, L+1, k), proj CQ (out, q*d_model), bias FP16 (out,)],+ rows = embedding then blocks 0..L-1; the router appends calibration FP16 (3,).+ The three matrices are CQ at HEAD_BITS with the model group size (the head+ trained on them under QAT); the runtime also accepts the earlier all-FP16+ layout with probes shaped (L+1, k, d_model). The walk consumes exactly+ extra - 1 tensors.CQ blob for a logical [out, in] matrix (in padded up to a multiple of group):first the packed indices (out * in_pad*bits/8 bytes), then the per-group L2@@ -65,7 +88,12 @@ field directly produces -1, 0, +1 in the NEON kernel. The codebook is notin the header: the 3-level Lloyd-Max centroids {-c, 0, +c} / sqrt(group),c = 1.2240064, are analytic.-RAW "tokenizer" blob (self-contained SentencePiece BPE dump; see RefTokenizer+BINARY (record bits=1, 2 levels) stores eight indices per byte. Its codebook is+also analytic: {-sqrt(2/pi), +sqrt(2/pi)} / sqrt(group).++RAW attachments follow the model tensors and heads. The first is the+self-contained SentencePiece BPE dump; the optional second is the speech+encoder's uncompressed NPZ archive. The SentencePiece dump format (see RefTokenizerbelow for the reference encoder/decoder this specifies):header u32 n_pieces, u32 pad/eos/bos/unk id, u8 add_dummy_prefix,u8 byte_fallback, u16 _pad;@@ -81,21 +109,21 @@ import numpy as npfrom .quantize import (_bits_for, _cq_codebook_np, _cq_hadamard_np,parse_bits_map, TERNARY_BITS)-from .architecture import TransformerConfig--HEAD_STAGES = ("conf-head", "tool-head")-TAG = 0x05E12A83+TAG = 0x05E12A84ALIGN = 64FP16, FP32, CQ, RAW = 1, 2, 3, 4CB_BITS = (2, 3, 4)+PACK_BITS = (1, *CB_BITS)TERNARY_RECORD_BITS = 5+IMAGE_MANIFEST_MAGIC = 7391.0+IMAGE_TENSOR_COUNT = 5TK_NORMAL, TK_UNKNOWN, TK_CONTROL, TK_USER_DEFINED, TK_BYTE = 0, 1, 2, 3, 4_TK_HDR = "<IIIIIBBH"_TK_REC = "<fBH"-_HDR_FMT = "<29If"+_HDR_FMT = "<48If"_REC_FMT = "<BBHIIIIQQII"REC_SIZE = struct.calcsize(_REC_FMT)@@ -103,25 +131,17 @@ REC_SIZE = struct.calcsize(_REC_FMT)def _geometry(config):from .architecture import head_dimsqk_hd, v_hd = head_dims(config)- if qk_hd != v_hd:+ if config.num_layers > 64:raise NotImplementedError(- f"cact format stores a single head_dim; split attention dims "- f"(qk {qk_hd} / v {v_hd}) need a format bump and the matching "- f"engine port before export")- if getattr(config, "lexicon_layers", ()):+ "cact global_mask holds 64 layers; deeper stacks need another bump")+ if len(getattr(config, "engram_layers", ())) > 16:raise NotImplementedError(- "cact format has no lexicon section; lexicon runs are "- "research-only until the format and engine support it")- if getattr(config, "sliding_window", 0):- raise NotImplementedError(- "cact header carries a single kv_window; the local/global layer "- "pattern needs the format bump and engine port before export")- head_dim = qk_hd+ "cact header holds 16 engram sites; denser engram needs another bump")orders = tuple(getattr(config, "engram_orders", (2, 3)))heads = getattr(config, "engram_heads", 0) or max(1, config.d_model // (len(orders) * ENGRAM_SUB_DIM))sub_dim = config.d_model // (len(orders) * heads)sites = tuple(getattr(config, "engram_layers", (2, 15)))- return head_dim, orders, heads, sub_dim, sites+ return qk_hd, v_hd, orders, heads, sub_dim, sitesENGRAM_SUB_DIM = 128@@ -182,8 +202,9 @@ def _unpack_ternary_crumbs(packed, in_pad):def _cq_pack(w, bits, group):- if bits not in CB_BITS and bits != TERNARY_BITS:- raise ValueError(f"CQ packing supports bits in {CB_BITS} or {TERNARY_BITS}; got bits={bits}")+ if bits not in PACK_BITS and bits != TERNARY_BITS:+ raise ValueError(+ f"CQ packing supports bits in {PACK_BITS} or {TERNARY_BITS}; got bits={bits}")cb = _cq_codebook_np(bits, group)H = _cq_hadamard_np(group)out, D = w.shape@@ -237,11 +258,15 @@ def _q(name, mat, bits, group):return _Tensor(name, CQ, mat.shape, packed.tobytes() + norms.tobytes(), group, rec_bits)-def _tensors(params, config, bits, group):- _, orders, heads, sub_dim, sites = _geometry(config)+def _tensors(params, config, bits, group, vocab_rows=None):+ _, _, orders, heads, sub_dim, sites = _geometry(config)num_tables = len(orders) * heads- ts = [_q("embedding", np.asarray(_get(params, ("embedding", "embedding"))), bits, group)]+ embedding = np.asarray(_get(params, ("embedding", "embedding")))+ if vocab_rows:+ embedding = np.concatenate([embedding[start:end] for start, end in vocab_rows])+ ts = [_q("embedding", embedding, bits, group)]+ taps_n = int(getattr(config, "qkv_conv_taps", 0))b = params["stack"]["layers"]["block"]sa, ha = b["self_attn"], b["hadamard_mlp"]for i in range(config.num_layers):@@ -250,6 +275,14 @@ def _tensors(params, config, bits, group):_q(f"layer{i:02d}.q_proj", np.asarray(sa["q_proj"]["kernel"][i]).T, bits, group),_q(f"layer{i:02d}.k_proj", np.asarray(sa["k_proj"]["kernel"][i]).T, bits, group),_q(f"layer{i:02d}.v_proj", np.asarray(sa["v_proj"]["kernel"][i]).T, bits, group),+ ]+ if taps_n:+ ts += [+ _fp16(f"layer{i:02d}.q_taps", sa["q_taps"][i]),+ _fp16(f"layer{i:02d}.k_taps", sa["k_taps"][i]),+ _fp16(f"layer{i:02d}.v_taps", sa["v_taps"][i]),+ ]+ ts += [_fp16(f"layer{i:02d}.q_norm", sa["q_norm"]["scale"][i]),_fp16(f"layer{i:02d}.k_norm", sa["k_norm"]["scale"][i]),_q(f"layer{i:02d}.gate_proj", np.asarray(sa["gate_proj"]["kernel"][i]).T, bits, group),@@ -259,7 +292,17 @@ def _tensors(params, config, bits, group):_fp16(f"layer{i:02d}.pre_hada", b["pre_hada_norm"]["scale"][i]),_fp16(f"layer{i:02d}.d1", ha["d1"][i]),_fp16(f"layer{i:02d}.d2", ha["d2"][i]),+ _fp16(f"layer{i:02d}.b2", ha["b2"][i]),_fp16(f"layer{i:02d}.d3", ha["d3"][i]),+ _fp16(f"layer{i:02d}.d4", ha["d4"][i]),+ _fp16(f"layer{i:02d}.w1a", ha["w1a"][i]),+ _fp16(f"layer{i:02d}.w1b", ha["w1b"][i]),+ _fp16(f"layer{i:02d}.w2a", ha["w2a"][i]),+ _fp16(f"layer{i:02d}.w2b", ha["w2b"][i]),+ _fp16(f"layer{i:02d}.w3a", ha["w3a"][i]),+ _fp16(f"layer{i:02d}.w3b", ha["w3b"][i]),+ _fp16(f"layer{i:02d}.cond_v", ha["cond_v"][i]),+ _fp16(f"layer{i:02d}.cond_u", ha["cond_u"][i]),]mhc = params["stack"]@@ -271,6 +314,15 @@ def _tensors(params, config, bits, group):L, nC, lanes = phi.shapets.append(_q(name, phi.transpose(0, 2, 1).reshape(L * lanes, nC), bits, group))+ from .architecture import _hada_perms+ hada_n = 1 << (config.d_model - 1).bit_length()+ split = bool(getattr(config, "ladder_widths", ()))+ p1, p2 = _hada_perms(hada_n, split)+ ts.append(_Tensor("hada_p1", FP32, (hada_n,),+ np.asarray(p1, np.float32).tobytes()))+ ts.append(_Tensor("hada_p2", FP32, (hada_n,),+ np.asarray(p2, np.float32).tobytes()))+for s in range(len(sites)):eg = params[f"engrams_{s}"]tables = np.asarray(eg["embedding"]).reshape(num_tables * config.engram_slots, sub_dim)@@ -282,28 +334,74 @@ def _tensors(params, config, bits, group):]ts.append(_fp16("final_norm", params["stack"]["final_norm"]["scale"]))- ts += _head_tensors(params)+ ts += _image_tensors(params, config, bits, group)+ ts += _head_tensors(params, group)return ts-HEAD_CODES = (("contrastive_head", 1), ("confidence_head", 2))---def _head_tensors(params):- present = [(k, c) for k, c in HEAD_CODES if k in params]+def _image_tensors(params, config, bits, group):+ sem_codes = int(getattr(config, "image_sem_codes", 0) or 0)+ if not sem_codes:+ return []+ code_offset = int(getattr(config, "image_code_offset", 0) or 0)+ positions = int(getattr(config, "image_positions", 256) or 256)+ sem_dim = int(getattr(config, "image_sem_dim", 0) or 0)+ tex_books = int(getattr(config, "image_tex_books", 0) or 0)+ tex_codes = int(getattr(config, "image_tex_codes", 0) or 0)+ tex_stages = int(getattr(config, "image_tex_stages", 0) or 0)+ tex_dim = int(getattr(config, "image_tex_dim", 0) or 0)+ if (code_offset < 0 or positions <= 0 or sem_dim <= 0 or tex_books <= 0+ or tex_codes <= 0 or tex_stages <= 0 or tex_dim <= 0+ or code_offset + sem_codes > int(config.vocab_size)):+ raise ValueError("invalid composite-image export geometry")++ semantic = np.asarray(+ params["image_sem_embedding"]["embedding"], np.float32)+ texture = np.asarray(+ params["image_texture_embedding"]["embedding"], np.float32)+ projection = np.asarray(params["image_projection"]["kernel"], np.float32)+ input_norm = np.asarray(params["image_input_norm"]["scale"], np.float32)+ composite = sem_dim + tex_books * tex_dim+ expected_texture = tex_books * tex_codes * tex_stages+ if semantic.shape != (sem_codes, sem_dim):+ raise ValueError("semantic image table disagrees with model config")+ if texture.shape != (expected_texture, tex_dim):+ raise ValueError("texture image table disagrees with model config")+ if projection.shape != (composite, int(config.d_model)):+ raise ValueError("image projection disagrees with composite geometry")+ if input_norm.shape != (int(config.d_model),):+ raise ValueError("image input norm disagrees with model dimension")++ manifest = np.asarray([+ IMAGE_MANIFEST_MAGIC, code_offset, positions, sem_codes, sem_dim,+ tex_books, tex_codes, tex_stages, tex_dim,+ ], np.float32)+ return [+ _Tensor("image.manifest", FP32, manifest.shape, manifest.tobytes()),+ _fp16("image.semantic", semantic),+ _fp16("image.texture", texture),+ _q("image_projection", projection.T, bits, group),+ _fp16("image.input_norm", input_norm),+ ]+++HEAD_MATRICES = ("probes", "query", "proj")+++def _head_tensors(params, group):+ from .architecture import HEADS+ from .quantize import HEAD_BITS+ present = [head for head in HEADS if head.key in params]if not present:return []- ts = [_fp16("heads.manifest", np.asarray([c for _, c in present], np.float16))]- for key, _ in present:- h = params[key]- kernel = np.asarray(h["proj"]["kernel"]).T- bias = (np.asarray(h["proj"]["bias"]) if "bias" in h["proj"]- else np.zeros(kernel.shape[0], np.float16))- ts += [- _fp16(f"{key}.probes", np.asarray(h["probes"])),- _fp16(f"{key}.proj", kernel),- _fp16(f"{key}.bias", bias),- ]+ ts = [_fp16("heads.manifest", np.asarray([head.code for head in present], np.float16))]+ for head in present:+ for name, value in head.export(params[head.key]):+ if name.rsplit(".", 1)[-1] in HEAD_MATRICES:+ matrix = np.asarray(value, np.float32).reshape(-1, value.shape[-1])+ ts.append(_q(name, matrix, HEAD_BITS, group))+ else:+ ts.append(_fp16(name, value))return ts@@ -319,7 +417,8 @@ def _tokenizer_blob(tok):sp = tok.spn = sp.GetPieceSize()markers = set(CHAT_MARKERS)- out = bytearray(struct.pack(_TK_HDR, n, PAD_ID, EOS_ID, BOS_ID, UNK_ID, 1, 1, 0))+ dummy = 1 if sp.IdToPiece(sp.Encode("a", out_type=int)[0]).startswith(_SP_META_SPACE) else 0+ out = bytearray(struct.pack(_TK_HDR, n, PAD_ID, EOS_ID, BOS_ID, UNK_ID, dummy, 1, 0))for i in range(n):piece = sp.IdToPiece(i)if sp.IsControl(i):@@ -337,25 +436,49 @@ def _tokenizer_blob(tok):return bytes(out)-def _pack_cact(params, config, bits, group, tokenizer, kv_window=0):+def vocab_rows_for(config):+ text = int(getattr(config, "out_vocab", 0) or 0)+ if not text or text >= int(config.vocab_size):+ return None+ return [(0, text)]+++def _pack_cact(params, config, bits, group, tokenizer, kv_window=0,+ speech_tokenizer=None, vocab_rows=None):params = {k: v for k, v in params.items()}- ts = _tensors(params, config, bits, group)+ ts = _tensors(params, config, bits, group, vocab_rows)+ vocab = (sum(end - start for start, end in vocab_rows) if vocab_rows+ else int(config.vocab_size))+ out_vocab = int(getattr(config, "out_vocab", 0) or 0)+ if out_vocab > vocab:+ raise ValueError(f"out_vocab {out_vocab} exceeds the exported vocab {vocab}")if tokenizer is not None:- if tokenizer.vocab_size != config.vocab_size:- raise ValueError(f"tokenizer vocab {tokenizer.vocab_size} != config {config.vocab_size}")+ if tokenizer.vocab_size > vocab:+ raise ValueError(f"tokenizer vocab {tokenizer.vocab_size} > exported vocab {vocab}")ts.append(_Tensor("tokenizer", RAW, (), _tokenizer_blob(tokenizer)))+ if speech_tokenizer is not None:+ ts.append(_Tensor("speech_tokenizer", RAW, (), speech_tokenizer))cb = np.concatenate([_cq_codebook_np(b, group) for b in CB_BITS]).astype(np.float32)kv_bits = int(getattr(config, "kv_bits", 8) or 8)- head_dim, orders, heads, sub_dim, sites = _geometry(config)+ qk_hd, v_hd, orders, heads, sub_dim, sites = _geometry(config)hada_n = 1 << (config.d_model - 1).bit_length()orders4 = (list(orders) + [0, 0, 0, 0])[:4]- sites4 = (list(sites) + [0, 0, 0, 0])[:4]+ sites16 = (list(sites) + [0] * 16)[:16]+ gmask = 0+ for g in getattr(config, "global_layers", ()) or ():+ gmask |= 1 << int(g)header = struct.pack(_HDR_FMT, TAG, len(ts), len(cb), int(kv_window or 0), kv_bits,- config.vocab_size, config.d_model, config.num_heads, config.num_kv_heads,- config.num_layers, head_dim, config.max_seq_len, hada_n,- config.mhc_lanes, config.engram_slots, sub_dim, len(orders) * heads,- ENGRAM_CONV_TAPS, max(orders), len(orders), *orders4, len(sites), *sites4,+ vocab, out_vocab,+ config.d_model, config.num_heads, config.num_kv_heads,+ config.num_layers, qk_hd, v_hd, config.max_seq_len, hada_n,+ config.mhc_lanes, int(getattr(config, "sliding_window", 0) or 0),+ gmask & 0xFFFFFFFF, (gmask >> 32) & 0xFFFFFFFF,+ int(getattr(config, "qkv_conv_taps", 0) or 0),+ config.engram_slots, sub_dim, len(orders) * heads,+ ENGRAM_CONV_TAPS, max(orders),+ int(getattr(config, "engram_seed_heads", 0) or 0),+ len(orders), *orders4, len(sites), *sites16,float(config.rope_theta)) + cb.tobytes()pos = len(header) + len(ts) * REC_SIZE@@ -379,15 +502,18 @@ def _pack_cact(params, config, bits, group, tokenizer, kv_window=0):def build_export(params, config, bits=4, group=128, tokenizer=None, bits_map=None,- kv_window=0):+ kv_window=0, speech_tokenizer=None, vocab_rows=None):b = parse_bits_map(bits_map) if bits_map else bits- return _pack_cact(params, config, b, group, tokenizer, kv_window)[0]+ return _pack_cact(params, config, b, group, tokenizer, kv_window,+ speech_tokenizer, vocab_rows)[0]def write_export(params, config, path, bits=4, group=128, tokenizer=None,- bits_map=None, kv_window=0):+ bits_map=None, kv_window=0, speech_tokenizer=None,+ vocab_rows=None):b = parse_bits_map(bits_map) if bits_map else bits- buf, n = _pack_cact(params, config, b, group, tokenizer, kv_window)+ buf, n = _pack_cact(params, config, b, group, tokenizer, kv_window,+ speech_tokenizer, vocab_rows)with open(path, "wb") as f:f.write(buf)return {"path": path, "bytes": len(buf), "tensors": n}@@ -400,15 +526,20 @@ def read_export(path):tag, num_tensors, cb_n, kv_window, kv_bits = hdr[:5]assert tag == TAG, "bad tag"geometry = dict(zip(- ("vocab_size", "d_model", "num_heads", "num_kv_heads", "num_layers",- "head_dim", "max_seq_len", "hada_n", "mhc_lanes", "engram_slots",- "engram_sub_dim", "num_engram_tables", "engram_conv_taps",- "engram_conv_dilation"), hdr[5:19]))- num_orders, orders4 = hdr[19], hdr[20:24]- num_sites, sites4 = hdr[24], hdr[25:29]+ ("vocab_size", "out_vocab", "d_model", "num_heads", "num_kv_heads",+ "num_layers", "qk_head_dim", "v_head_dim", "max_seq_len", "hada_n",+ "mhc_lanes", "sliding_window", "global_mask_lo", "global_mask_hi",+ "qkv_conv_taps", "engram_slots", "engram_sub_dim",+ "num_engram_tables", "engram_conv_taps", "engram_conv_dilation",+ "engram_seed_heads"), hdr[5:26]))+ gmask = geometry.pop("global_mask_lo") | (geometry.pop("global_mask_hi") << 32)+ geometry["global_layers"] = tuple(i for i in range(geometry["num_layers"])+ if gmask >> i & 1)+ num_orders, orders4 = hdr[26], hdr[27:31]+ num_sites, sites16 = hdr[31], hdr[32:48]geometry["engram_orders"] = tuple(orders4[:num_orders])- geometry["engram_layers"] = tuple(sites4[:num_sites])- geometry["rope_theta"] = hdr[29]+ geometry["engram_layers"] = tuple(sites16[:num_sites])+ geometry["rope_theta"] = hdr[48]off = struct.calcsize(_HDR_FMT)codebook = np.frombuffer(raw[off:off + cb_n * 4], np.float32).copy()off += cb_n * 4@@ -531,36 +662,3 @@ class RefTokenizer:if self.add_dummy and text.startswith(" "):text = text[1:]return text---def main(args):- from .run import load_checkpoint- from .tokenizer import get_tokenizer- params, config, run_meta = load_checkpoint(args.checkpoint, return_run=True)- out = getattr(args, "out", None) or "needle.cact"- bits = getattr(args, "bits", None)- bits_map = getattr(args, "bits_map", None)- if bits is None and not bits_map:- bits_map = getattr(config, "weight_bits", "") or ""- if bits_map:- print(f"[export] scheme from the checkpoint: {bits_map}")- elif (run_meta or {}).get("stage") in ("qapt",) + HEAD_STAGES:- raise ValueError(- f"{os.path.basename(args.checkpoint)} is a "- f"'{run_meta['stage']}' checkpoint with no config.weight_bits. "- "Exporting it would silently ship plain W4 - numerics it was "- "never post-trained for. Migrate it "- "(scripts/migrate_checkpoint_numerics.py) or pass --bits-map.")- else:- bits = 4- elif bits is None:- bits = 4- from .architecture import effective_kv_window- kv_window = effective_kv_window(config)- info = write_export(params, config, out, bits=bits, bits_map=bits_map,- tokenizer=get_tokenizer(config.vocab_size),- kv_window=kv_window)- mb = info["bytes"] / 1e6- w = f"mixed[{bits_map}]" if bits_map else f"W{bits}"- print(f"wrote {info['path']}: {info['tensors']} tensors (+tokenizer), "- f"{mb:.2f} MB ({w}A8)")diff --git a/needle/model/finetune.py b/needle/model/finetune.pyindex 5eca39c..b2cfaad 100644--- a/needle/model/finetune.py+++ b/needle/model/finetune.py@@ -28,7 +28,7 @@ def _training_rng(seed):return np.random.default_rng(int(seed))-DEFAULT_BASE = "checkpoints/needle2.pkl"+DEFAULT_BASE = "checkpoints/needle3.safetensors"OPENROUTER_URL = os.environ.get("OPENROUTER_URL", "https://openrouter.ai/api/v1/chat/completions"@@ -258,6 +258,19 @@ def load_jsonl(path, tokenizer, max_len):return np.array(seqs, np.int32), np.array(masks, np.float32)+def rung(params, config, layers):+ from .architecture import ladder_config, ladder_slice++ fields = dict(config.__dict__)+ fields.update(ladder_depths=(), ladder_sample=False, ladder_widths=())+ config = type(config)(**fields)+ if not layers or layers == config.num_layers:+ return params, config+ if not 2 <= layers < config.num_layers:+ raise ValueError(f"--layers must be in [2, {config.num_layers}], got {layers}")+ return ladder_slice(params, config, layers), ladder_config(config, layers)++def lora_target_paths(params):import jax.numpy as jnpfrom flax.traverse_util import flatten_dict@@ -319,6 +332,8 @@ def finetune_local(args, progress=None):workers=getattr(args, "workers", 8))params, config = load_checkpoint(base_path)+ layers = getattr(args, "layers", None)+ params, config = rung(params, config, layers)config.dtype = "float32"params = jax.tree.map(lambda a: np.asarray(a).astype(np.float32), params)backend = jax.default_backend().lower()@@ -328,6 +343,7 @@ def finetune_local(args, progress=None):config.scan_unroll = config.num_layersparams = jax.device_put(params)emit(f" {'backend':<9} {backend} float32")+ emit(f" {'depth':<9} {config.num_layers} layers")tokenizer = get_tokenizer(config.vocab_size)max_len = fit_max_len(data_path, tokenizer, args.max_len)seqs, masks = load_jsonl(data_path, tokenizer, max_len)@@ -437,6 +453,7 @@ def finetune_local(args, progress=None):"qat_bits": qat_bits,"qat_bits_map": qat_bits_map,"seed": seed,+ "layers": int(config.num_layers),})print(f" {'adapter':<9} {out}")print(f" {'next':<9} needle build {base_path} --lora {out}")@@ -447,14 +464,23 @@ def build_main(args):import jax.numpy as jnpfrom .run import load_checkpointfrom .architecture import effective_kv_window- from .export import write_export+ from .export import vocab_rows_for, write_exportparams, config, _ = load_checkpoint(args.checkpoint, return_run=True)+ layers = getattr(args, "layers", None)adapter_qat_bits = Noneadapter_qat_bits_map = Noneif args.lora:adapter = read_adapter(args.lora)+ adapter_layers = adapter.get("layers")+ if adapter_layers and layers and int(layers) != int(adapter_layers):+ raise ValueError(+ f"adapter was trained on the {adapter_layers}-layer rung, but --layers "+ f"{layers} would export a different depth")+ layers = adapter_layers or layers+ params, config = rung(params, config, layers)+ if args.lora:lora = {tuple(key.split("/")): {"A": jnp.asarray(v["A"]), "B": jnp.asarray(v["B"])}for key, v in adapter["lora"].items()}params = merge_lora(params, lora, adapter["scale"])@@ -488,7 +514,8 @@ def build_main(args):bits=int(bits) if bits else 4,bits_map=bits_map,tokenizer=get_tokenizer(config.vocab_size),- kv_window=effective_kv_window(config))+ kv_window=effective_kv_window(config),+ vocab_rows=vocab_rows_for(config))scheme = f"mixed[{bits_map}]" if bits_map else f"W{bits}"print(f" {'wrote':<9} {info['path']} {info['bytes'] / 1e6:.2f} MB {info['tensors']} tensors {scheme}A8")print(f" {'next':<9} needle.Needle(weights={out!r}, tools=[...])")diff --git a/needle/model/quantize.py b/needle/model/quantize.pyindex fd7a487..9013c46 100644--- a/needle/model/quantize.py+++ b/needle/model/quantize.py@@ -7,6 +7,14 @@ import jaximport jax.numpy as jnp+def _leaf_key(path):+ """Return the owning parameter key, skipping Flax metadata wrappers."""+ for entry in reversed(path):+ if hasattr(entry, "key"):+ return str(entry.key)+ return ""++def fake_quant(w, group_size=128, bits=4):qmax = 2 ** (bits - 1) - 1D = w.shape[-1]@@ -22,14 +30,6 @@ def fake_quant(w, group_size=128, bits=4):return w + jax.lax.stop_gradient(q - w)-def quantize_params(params, group_size=128, bits=4):- def q(path, leaf):- name = path[-1].key- if name in ("kernel", "embedding") and leaf.ndim >= 2:- return fake_quant(leaf, group_size, bits)- return leaf- return jax.tree_util.tree_map_with_path(q, params)-def fake_quant_act(x):return fake_quant(x, x.shape[-1], ACT_BITS)@@ -38,11 +38,25 @@ def cq_fake_quant_kv(x, bits, group=64):return x + jax.lax.stop_gradient(cq_quantize(x, bits, group) - x)+def a8_fake_quant_kv(x):+ """Per-head symmetric A8 KV quantization matching the native cache."""+ return fake_quant(x, x.shape[-1], 8)+++def maybe_quant_query(x, quant):+ """Per-head A8 query quantization matching the native attention kernel."""+ if not ACT_BITS:+ return x+ return jax.lax.cond(quant, a8_fake_quant_kv, lambda t: t, x)++def maybe_quant_kv(x, quant):if not KV_BITS:return x+ quantize = (a8_fake_quant_kv if KV_BITS >= 8 else+ lambda t: cq_fake_quant_kv(t, KV_BITS, _KV_GROUP))return jax.lax.cond(- quant, lambda t: cq_fake_quant_kv(t, KV_BITS, _KV_GROUP), lambda t: t, x)+ quant, quantize, lambda t: t, x)QAT_EVERY = 0@@ -60,7 +74,7 @@ def configure_qat(every, weight_group=128, weight_bits=4):def configure_deploy(act_bits=8, kv_bits=8, kv_group=64):global ACT_BITS, KV_BITS, _KV_GROUP- kv_bits = 0 if int(kv_bits) >= 8 else int(kv_bits)+ kv_bits = int(kv_bits)changed = (ACT_BITS, KV_BITS, _KV_GROUP) != (int(act_bits), int(kv_bits), int(kv_group))ACT_BITS, KV_BITS, _KV_GROUP = int(act_bits), int(kv_bits), int(kv_group)if changed:@@ -68,7 +82,7 @@ def configure_deploy(act_bits=8, kv_bits=8, kv_group=64):def quantize_params_configured(params):- return quantize_params(params, _WEIGHT_GROUP, _WEIGHT_BITS)+ return cq_ste_params(params, _WEIGHT_BITS, _WEIGHT_GROUP)def deploy_quantize(params, config):@@ -93,6 +107,9 @@ def maybe_quant_weights(params, do_quantize):@functools.lru_cache(maxsize=None)def _lloyd_max_gaussian(bits, iters=200, samples=400000, seed=0):+ if bits == 1:+ c = math.sqrt(2.0 / math.pi)+ return np.array([-c, c])levels = 1 << bitsx = np.sort(np.random.RandomState(seed).randn(samples))c = x[((np.arange(levels) + 0.5) / levels * samples).astype(int)].astype(np.float64)@@ -148,28 +165,68 @@ def cq_quantize(w, bits, group_size=128, codebook=None):def _is_quant_leaf(path, leaf):- key = path[-1].key+ key = _leaf_key(path)return ((key in ("kernel", "embedding") or key.startswith("mhc_phi"))and getattr(leaf, "ndim", 0) >= 2)def _reduces_second_last(path):- key = path[-1].key+ key = _leaf_key(path)return key == "kernel" or key.startswith("mhc_phi")def cq_quantize_params(params, bits, group_size=128):- cb = jnp.asarray(_cq_codebook_np(bits, group_size))+ return _map_quant_leaves(+ params, lambda w, i: cq_quantize(w, bits, group_size))- def q(path, leaf):- if not _is_quant_leaf(path, leaf):- return leaf- if _reduces_second_last(path):- rotated = cq_quantize(jnp.swapaxes(leaf, -1, -2), bits, group_size, cb)- return jnp.swapaxes(rotated, -1, -2)- return cq_quantize(leaf, bits, group_size, cb)- return jax.tree_util.tree_map_with_path(q, params)+AB_KEY = "ab_scales"+++def _ab_of(params):+ return params.get(AB_KEY) if hasattr(params, "get") else None+++def ab_init_params(params, seed=0, signs=False):+ """a = b = 1 (exact no-op at step 0; the PTQ default) or, with signs=True,+ a = b = random +-1 (randomized-Hadamard init for from-scratch/SFT runs).+ a*b = 1 either way, so effective FP weights are unchanged. A tree that+ already carries AB (a resumed QAPT checkpoint) is returned unchanged."""+ if _ab_of(params):+ return dict(params)+ rng = np.random.RandomState(seed)+ ab = {}++ def visit(path, leaf):+ if _is_quant_leaf(path, leaf):+ r = leaf.shape[-2] if _reduces_second_last(path) else leaf.shape[-1]+ v = ((rng.randint(0, 2, r) * 2 - 1) if signs else np.ones(r))+ ab[leaf_name(path)] = {"a": jnp.asarray(v, jnp.float32),+ "b": jnp.asarray(v, jnp.float32)}+ return leaf++ jax.tree_util.tree_map_with_path(visit, params)+ out = dict(params)+ out[AB_KEY] = ab+ return out+++def freeze_quant_grads(grads):+ """Zero the gradients of every CQ-quantized leaf (signs stay frozen);+ norms, gates, AB scales and other smooth params keep theirs."""+ def z(path, leaf):+ return jnp.zeros_like(leaf) if _is_quant_leaf(path, leaf) else leaf+ return jax.tree_util.tree_map_with_path(z, grads)+++def ab_fold_params(params):+ """FP-effective tree of an AB-carrying model (w <- a*b*w, AB_KEY dropped):+ the clean-forward reference of the function the quant transforms deploy."""+ if not _ab_of(params):+ return params+ folded = dict(_map_quant_leaves(params, lambda w, i: w))+ folded.pop(AB_KEY)+ return foldeddef cq_model_bytes(params, bits, group_size=128):@@ -196,7 +253,7 @@ def model_bytes_fp16(params):CQ_GROUP_SIZE = 128-CQ_BITS = (2, 3, 4)+CQ_BITS = (1, 2, 3, 4)MIN_BITS, MAX_BITS = 1, 8@@ -252,6 +309,9 @@ def add_cq_noise(w, key, scale, group_size=CQ_GROUP_SIZE):def _map_quant_leaves(params, fn):+ """fn(leaf_with_reduction_axis_last, index) over the CQ-quantized leaves;+ AB scales, when present, sandwich fn: leaf -> a * fn(b * leaf)."""+ ab = _ab_of(params)counter = [0]def q(path, leaf):@@ -259,9 +319,15 @@ def _map_quant_leaves(params, fn):return leafi = counter[0]counter[0] += 1+ s = ab.get(leaf_name(path)) if ab else None++ def apply(w):+ out = fn(w * s["b"] if s else w, i)+ return out * s["a"] if s else out+if _reduces_second_last(path):- return jnp.swapaxes(fn(jnp.swapaxes(leaf, -1, -2), i), -1, -2)- return fn(leaf, i)+ return jnp.swapaxes(apply(jnp.swapaxes(leaf, -1, -2)), -1, -2)+ return apply(leaf)return jax.tree_util.tree_map_with_path(q, params)@@ -283,7 +349,13 @@ def noise_params_only(params, key, scale, name, group_size=CQ_GROUP_SIZE):def leaf_name(path):- return "/".join(getattr(p, "key", str(p)) for p in path)+ # Partitioned/AxisMetadata leaves add a trailing GetAttrKey("value").+ # That is container metadata, not part of the checkpoint parameter name.+ return "/".join(+ str(p.key) if hasattr(p, "key") else str(p.idx)+ for p in path+ if hasattr(p, "key") or hasattr(p, "idx")+ )def quant_leaf_names(params):@@ -355,6 +427,20 @@ def cq_ste(w, bits, group_size=CQ_GROUP_SIZE):return w + jax.lax.stop_gradient(cq_quantize(w, bits, group_size) - w)+HEAD_BITS = 4+++def head_weight(w, quant, reduce_second_last=False, group_size=CQ_GROUP_SIZE):+ """A probe-head matrix as the engine sees it: CQ at HEAD_BITS under QAT,+ straight-through for the gradient; kernels quantize along their input axis."""+ def fake(x):+ if reduce_second_last:+ return jnp.swapaxes(+ cq_ste(jnp.swapaxes(x, -1, -2), HEAD_BITS, group_size), -1, -2)+ return cq_ste(x, HEAD_BITS, group_size)+ return jax.lax.cond(quant, fake, lambda x: x, w)++def cq_ste_params(params, bits, group_size=CQ_GROUP_SIZE):return _map_quant_leaves(params, lambda w, i: cq_ste(w, bits, group_size))@@ -364,3 +450,48 @@ def cq_ste_mixed_params(params, bits_map, default_bits, group_size=CQ_GROUP_SIZEreturn _map_quant_leaves(params,lambda w, i: cq_ste(w, _bits_for(names[i], bits_map, default_bits), group_size))+++def kurtosis_penalty(params, group_size=None):+ g = _WEIGHT_GROUP if group_size is None else group_size+ ab = _ab_of(params)+ H = jnp.asarray(_cq_hadamard_np(g))+ terms = []++ def visit(path, leaf):+ if not _is_quant_leaf(path, leaf):+ return leaf+ s = ab.get(leaf_name(path)) if ab else None+ w = jnp.swapaxes(leaf, -1, -2) if _reduces_second_last(path) else leaf+ if s is not None:+ w = w * s["b"]+ pad = (-w.shape[-1]) % g+ wp = jnp.pad(w, [(0, 0)] * (w.ndim - 1) + [(0, pad)]) if pad else w+ rot = wp.reshape(*wp.shape[:-1], -1, g).astype(jnp.float32) @ H+ m2 = jnp.mean(rot ** 2, axis=-1)+ m4 = jnp.mean(rot ** 4, axis=-1)+ kurt = m4 / jnp.maximum(m2 ** 2, 1e-12)+ dev = (kurt - 3.0) ** 2+ terms.append((jnp.sum(dev), dev.size))+ return leaf++ jax.tree_util.tree_map_with_path(visit, params)+ return sum(t for t, _ in terms) / max(sum(n for _, n in terms), 1)+++def attractor_penalty(params, bits=None, group_size=None):+ b = _WEIGHT_BITS if bits is None else bits+ g = _WEIGHT_GROUP if group_size is None else group_size+ terms = []++ def fn(w, i):+ q = jax.lax.stop_gradient(cq_quantize(w, b, g))+ d = (w - q).astype(jnp.float32)+ ref = jax.lax.stop_gradient(w.astype(jnp.float32))+ terms.append((jnp.sum(d * d), jnp.sum(ref * ref)))+ return w++ _map_quant_leaves(params, fn)+ num = sum(t for t, _ in terms)+ den = sum(r for _, r in terms)+ return num / jnp.maximum(den, 1e-12)diff --git a/needle/model/run.py b/needle/model/run.pyindex c5e92ca..2784fb4 100644--- a/needle/model/run.py+++ b/needle/model/run.py@@ -90,12 +90,11 @@ def load_checkpoint(path, return_run=False):f"(got format_version={version!r}). Old encoder-decoder/tool-calling "f"checkpoints are incompatible with this branch.")- config = ckpt["config"]- if isinstance(config, dict):- config = TransformerConfig(**config)+ config = TransformerConfig.from_saved(ckpt["config"])+ params = {k: v for k, v in ckpt["params"].items() if not k.startswith("mtp_")}if return_run:- return ckpt["params"], config, ckpt.get("run") or {}- return ckpt["params"], config+ return params, config, ckpt.get("run") or {}+ return params, configdef _get_decode_fn(model, buf_len):diff --git a/needle/model/tokenizer.py b/needle/model/tokenizer.pyindex 83d7a37..9200768 100644--- a/needle/model/tokenizer.py+++ b/needle/model/tokenizer.py@@ -19,16 +19,34 @@ TOOL_CALL_START = "<tool_call>"TOOL_CALL_END = "</tool_call>"TOOL_RESULT_START = "<tool_result>"TOOL_RESULT_END = "</tool_result>"+CONTEXT_START = "<context>"+CONTEXT_END = "</context>"+IMAGE_START = "<image>"+IMAGE_END = "</image>"+SPEECH_START = "<speech>"+SPEECH_END = "</speech>"+TTS_START = "<tts>"+TTS_END = "</tts>"+IMAGEN_START = "<imagen>"+IMAGEN_END = "</imagen>"+EXTRACT_START = "<extract>"+EXTRACT_END = "</extract>"+SCHEMA_START = "<schema>"+SCHEMA_END = "</schema>"CHAT_MARKERS = [IM_START, IM_END, THINK_START, THINK_END,TOOLS_START, TOOLS_END, TOOL_CALL_START, TOOL_CALL_END,TOOL_RESULT_START, TOOL_RESULT_END,+ CONTEXT_START, CONTEXT_END, IMAGE_START, IMAGE_END,+ SPEECH_START, SPEECH_END,+ TTS_START, TTS_END, IMAGEN_START, IMAGEN_END,+ EXTRACT_START, EXTRACT_END, SCHEMA_START, SCHEMA_END,]IM_START_ID, IM_END_ID, THINK_START_ID, THINK_END_ID = 4, 5, 6, 7(TOOLS_START_ID, TOOLS_END_ID, TOOL_CALL_START_ID, TOOL_CALL_END_ID,TOOL_RESULT_START_ID, TOOL_RESULT_END_ID) = range(8, 14)-HF_REPO = "Cactus-Compute/needle2"+HF_REPO = "Cactus-Compute/needle3"_HF_TOKENIZER_DIR = "tokenizer"@@ -61,6 +79,10 @@ class SANTokenizer:def vocab_size(self):return self.sp.GetPieceSize()+ def piece_id(self, piece):+ i = self.sp.PieceToId(piece)+ return i if i > 0 else None+def encode(self, text):return self.sp.Encode(text, out_type=int)@@ -110,4 +132,9 @@ def get_tokenizer(vocab_size=None):f"No pretraining tokenizer at {model_path} and HF download failed ({e}). "f"It ships in the {HF_REPO} Hugging Face repo; retry once it is reachable.") from e- return SANTokenizer(model_path)+ tokenizer = SANTokenizer(model_path)+ if vocab_size is not None and vocab_size < tokenizer.vocab_size:+ raise ValueError(+ f"Tokenizer at {model_path} has {tokenizer.vocab_size} pieces, the model "+ f"vocabulary holds {vocab_size}")+ return tokenizerdiff --git a/tests/conftest.py b/tests/conftest.pyindex 7b2c715..ca183dc 100644--- a/tests/conftest.py+++ b/tests/conftest.py@@ -49,9 +49,10 @@ def tiny_checkpoint(tmp_path_factory):from needle.model.architecture import SimpleAttentionNetwork, TransformerConfigconfig = TransformerConfig(- vocab_size=8192, d_model=64, num_heads=4, num_kv_heads=2, num_layers=2,- max_seq_len=128, engram_layers=(1,), engram_slots=64, mhc_lanes=2,- flash=False,+ vocab_size=8192, out_vocab=8192, d_model=64, num_heads=4, num_kv_heads=2,+ num_layers=4, qk_head_dim=16, v_head_dim=16, max_seq_len=128,+ engram_layers=(1, 3), engram_slots=64, global_layers=(3,), sliding_window=32,+ mhc_lanes=2, qkv_conv_taps=3, flash=False,)model = SimpleAttentionNetwork(config)params = model.init(jax.random.PRNGKey(0), jnp.ones((1, 8), jnp.int32))["params"]diff --git a/tests/test_build.py b/tests/test_build.pyindex 049e498..11f134a 100644--- a/tests/test_build.py+++ b/tests/test_build.py@@ -26,6 +26,33 @@ def test_build_exports_loadable_cact(tiny_checkpoint, tmp_path):assert any(isinstance(t, (bytes, bytearray)) for t in tensors)+def test_build_layers_exports_the_rung(tiny_checkpoint, tmp_path):+ from needle.model.finetune import build_main+ from needle.model.export import read_export++ out = str(tmp_path / "tiny_3l.cact")+ args = _build_args(tiny_checkpoint, out, bits="4")+ args.layers = 3+ build_main(args)+ header, _ = read_export(out)+ assert header["num_layers"] == 3+++def test_load_checkpoint_drops_the_mtp_block(tiny_checkpoint, tmp_path):+ import pickle+ import numpy as np+ from needle.model.run import load_checkpoint++ with open(tiny_checkpoint, "rb") as handle:+ checkpoint = pickle.load(handle)+ checkpoint["params"]["mtp_combine"] = {"kernel": np.ones((4, 4), np.float32)}+ path = tmp_path / "with_mtp.pkl"+ with open(path, "wb") as handle:+ pickle.dump(checkpoint, handle)+ params, _ = load_checkpoint(str(path))+ assert "mtp_combine" not in params++def test_build_at_two_bits(tiny_checkpoint, tmp_path):from needle.model.finetune import build_mainfrom needle.model.export import read_exportdiff --git a/tests/test_finetune.py b/tests/test_finetune.pyindex e45a8a7..6c67a53 100644--- a/tests/test_finetune.py+++ b/tests/test_finetune.py@@ -116,6 +116,35 @@ def test_auto_qat_preserves_checkpoint_mixed_bit_map(tiny_checkpoint, tmp_path):upload=False, bits="4"))+def test_finetune_on_a_rung_builds_that_depth(tiny_checkpoint, tmp_path):+ from needle.model.finetune import finetune_local, build_main+ from needle.model.checkpoints import read_adapter+ from needle.model.export import read_export++ data = tmp_path / "data.jsonl"+ _write_data(data)+ adapter_path = tmp_path / "rung.safetensors"+ args = _finetune_args(data, tiny_checkpoint, adapter_path, tmp_path / "ck", qat_bits="none")+ args.layers = 2+ progress = []+ finetune_local(args, progress=progress.append)+ assert any("2 layers" in message for message in progress)+ adapter = read_adapter(adapter_path)+ assert adapter["layers"] == 2+ assert all(v["A"].shape[0] == 2 for v in adapter["lora"].values())++ out = str(tmp_path / "rung.cact")+ build_main(types.SimpleNamespace(checkpoint=tiny_checkpoint, lora=str(adapter_path),+ out=out, upload=False, bits="4"))+ header, _ = read_export(out)+ assert header["num_layers"] == 2++ with pytest.raises(ValueError, match="2-layer rung"):+ build_main(types.SimpleNamespace(checkpoint=tiny_checkpoint, lora=str(adapter_path),+ out=str(tmp_path / "wrong.cact"), upload=False,+ bits="4", layers=3))++def test_finetune_rng_is_controlled_by_seed():from needle.model.finetune import _training_rngdiff --git a/tests/test_lora.py b/tests/test_lora.pyindex 91b0f7f..51ff317 100644--- a/tests/test_lora.py+++ b/tests/test_lora.py@@ -11,7 +11,7 @@ def test_lora_target_paths_selects_projection_kernels():"gate_proj": {"kernel": jnp.ones((2, 8, 8))},"q_norm": {"scale": jnp.ones((2, 8))},}}}, "final_norm": {"scale": jnp.ones((8,))}},- "mtp_combine": {"kernel": jnp.ones((8, 8))},+ "confidence_head": {"proj": {"kernel": jnp.ones((8, 8))}},}paths = lora_target_paths(params)assert {p[-2] for p in paths} == {"q_proj", "gate_proj"}