Skip to content

FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.

2.0 KB
Raw Blame
"""Qwen3.8-Flash-Next (model_type qwen4_exp); image input rides on the shared Qwen VL tower.

48 decoder layers on hc_count=4 hyper-connection residual streams R [T, 4*hidden]:
embed -> repeat(1, 4) -> [PLE at zero-based layer 1] -> per layer attn_hc.mix -> (GDN | QSA) -> attn_hc.combine -> mlp_hc.mix -> MoE -> mlp_hc.combine -> top-level mixer.mix -> lm_head.
Layer contract: forward(R [T, 4*hidden], batch) -> R' [T, 4*hidden].

Contracts shared across modules (do not rename):
- The PLE dilated-conv left context lives on the LinearStatePool slots as the declared slot state ``ple_conv`` (config.ple_slot_states -> ModelConfig.slot_states), read back with ``pool.slot_state("ple_conv", layer_id)``; same slot / COW / snapshot lifecycle as conv_states / recurrent_states.
- kvcache.qsa_pool.QSAKVCache(MHAKVCache): ``cmp_k_cache(slot) -> [rows, index_head_dim]`` (compressed index keys, row = kv slot // index_ratio), ``pending_ring(slot) -> [num_req_slots, ring_capacity, index_head_dim]`` (per-request pre-RoPE index-k tail indexed by table_idx, never cleared), ``cmp_scratch_base`` (int, first scratch row for non-closing decode writes). ``slot`` is the sparse layer's order in the attention backend.
"""

from .config import parse_config
from .gguf import gguf_mm_config, iter_gguf_weights, parse_gguf_config, to_gguf
from .model import Qwen4ExpForCausalLM, Qwen4ExpForConditionalGeneration
from .weight import (
    ftw_side_files,
    nvfp4_expert_spec,
    iter_vision_weights,
    iter_weights,
    load_ple_table,
)

# Official FP8 checkpoints share qwen3_5_moe's block-fp8 expert layout (same
# model.language_model.layers.* keys), so reuse its expert reader.
from freetoken.models.qwen3_5_moe.weight import iter_expert_pieces

__all__ = [
    "ftw_side_files",
    "nvfp4_expert_spec",
    "Qwen4ExpForCausalLM",
    "Qwen4ExpForConditionalGeneration",
    "iter_vision_weights",
    "iter_weights",
    "load_ple_table",
    "parse_config",
    "iter_expert_pieces",
    "gguf_mm_config",
    "iter_gguf_weights",
    "parse_gguf_config",
    "to_gguf",
]