Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++
774 B
Raw Blame
{
  "comment": "a quick Maya-S-like test model (~200 GB): ggml's IQ2_XXS quantizer runs at ~1 M weights/s a thread (a full Maya-S takes ~8 h on 12 threads), so most layers take Q2_K / Q3_K (20-80 M/s) and a few layers each low-bit type the engine reads, on real weights",
  "experts": {"gate": "Q2_K", "up": "Q2_K", "down": "Q3_K"},
  "layers": [
    {"layers": "10", "gate": "IQ2_XXS", "up": "IQ2_XXS", "down": "IQ3_XXS"},
    {"layers": "20", "gate": "IQ4_XS", "up": "IQ4_XS", "down": "Q4_K"},
    {"layers": "25", "gate": "IQ3_XXS", "up": "IQ3_XXS", "down": "IQ3_S"},
    {"layers": "30", "gate": "IQ2_S", "up": "IQ2_S", "down": "IQ3_S"}
  ],
  "dense": "Q8_0",
  "dense_keep": ["hc_", "engram", "norm", "attn_sinks", "exp_probs_b", "ffn_gate_inp"],
  "engram": "skip"
}