Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.
C++
trellis: README, column-profile check of the Gyro error
Niko12216f99190f8260
diff --git a/tools/trellis/README.md b/tools/trellis/README.mdnew file mode 100644index 000000000..a76354aa7--- /dev/null+++ b/tools/trellis/README.md@@ -0,0 +1,34 @@+# Trellis encoder (research, branch f-dsv41-enc)++An encoder for the trellis expert formats that Strata already decodes (TQ2_T, TQK6, TQK7 of agentionai's llama.cpp fork)+and for two experimental lower rates, TQK5X (22 bytes per 128 weights, 1.375 bpw) and TQK4X (18 bytes, 1.125 bpw).+Written for the DeepSeek-V4.1-Flash compression plan (gate 2 and the start of gate 3). Nothing here changes an existing+type, layout or default; the engine's CPU decoder only gains K = 5 and 4 (ids 224 / 225, experimental version 1).++| file | what |+| --- | --- |+| `tqenc.hpp`, `tqenc.cpp` | the encoder: tail-biting Viterbi on the 2^15-state shift trellis, fp16 scale refinement, importance weights; a C API |+| `tqenc_test.cpp` | the tests (about 64,000 checks): streams and the circular wrap, Viterbi against a naive DP, planted solutions, bit-exact decode against `tq_states.hpp` and the scalar / AVX2 / AVX-512 kernels, special blocks, thread determinism, quality against the rate-distortion limit |+| `tqenc_hip.cpp`, `tqenc_hip_test.cpp` | the same search on a GPU (HIP, gfx1201): bit-identical blocks, about 20 M weights/s on one R9700 |+| `../tqenc.py` | Python wrapper (ctypes): `encode`, `decode`, the input-only block-128 Hadamard, signs, the `.tqx` container |+| `pilot.py`, `pilot_summary.py` | the block-quality pilot on a source GGUF (local files or HTTP range requests) |+| `gyro_reencode.py`, `gyro_vs_original.py`, `gyro_error_profile.py`, `hf_slice.py` | checks against agentionai's Gyro files and the original Qwen weights |++Build and test (Linux x86-64, g++ with AVX2; the GPU part needs ROCm):++ sh tools/trellis/build.sh test && build-tqenc/tqenc_test # --quick skips the quality table+ sh tools/trellis/build.sh lib && python3 tools/trellis/test_tqenc_py.py+ sh tools/trellis/build.sh hip && HIP_VISIBLE_DEVICES=3 build-tqenc/tqenc_hip_test++`build.sh hip` compiles `tqenc.cpp` with g++ on purpose: under hipcc, `ggml-common-tq.h` turns its codebook into a+`__device__` variable and the host side reads zeros.++## Format facts the encoder relies on (checked against the decoder, not the comments)++* A block is `{fp16 d; 4K bytes}` for 128 weights: 32 steps, step t gives weights 4t..4t+3.+* State of step t: 15 bits. TQK: the 16-bit window of the stream at bit `(31 - t) K` (LSB first, circular modulo 32 K bits), masked+ to 15 bits. TQ2_T: `(qs[(t+31)%32] << 8 | qs[t]) & 0x7fff`.+* Weights of a step: `x = state * 0x9e3779b1`, the two codebook points `lut[x >> 21]` and `lut[(x >> 10) & 2047]`, times d. The+ decoder's table is `round(fp16 * 8192)`; the encoder uses that table, so its reconstruction is the decoder's, bit for bit.+* A legal stream is any byte string (every state is a window of the same circular stream); tail-biting only matters to the search.+* K5 and K4 are the TQK rule with K = 5 and 4. Nothing else changes (hash, codebook, mask, bit order, scale).diff --git a/tools/trellis/gyro_column_profile.py b/tools/trellis/gyro_column_profile.pynew file mode 100644index 000000000..cd212dc27--- /dev/null+++ b/tools/trellis/gyro_column_profile.py@@ -0,0 +1,70 @@+"""gyro_column_profile.py - is the unevenness of agentionai's per-input-column error a property of the layer (an+activation-aware objective shared by all experts that read the same input) or noise? Compares the per-column squared-error+profile (rotated basis) between experts and between gate and up of one layer, for their file and for our plain encode.+"""+from __future__ import annotations++import argparse+import mmap+import pathlib+import sys++import numpy as np++HERE = pathlib.Path(__file__).resolve().parent+sys.path.insert(0, str(HERE.parent))+sys.path.insert(0, str(HERE))+import gguf_reader # noqa: E402+import hf_slice # noqa: E402+import tqenc # noqa: E402+++def main() -> int:+ ap = argparse.ArgumentParser()+ ap.add_argument("--gguf", required=True)+ ap.add_argument("--repo", default="Qwen/Qwen3.8-Flash-Next")+ ap.add_argument("--layer", type=int, default=10)+ ap.add_argument("--experts", default="0,1,2,3")+ a = ap.parse_args()+ g = gguf_reader.GGUFFile(a.gguf)+ fh = open(a.gguf, "rb")+ mm = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ)+ by = {t.name: t for t in g.tensors}+ sw = [int(x) for x in g.metadata["prism.hadamard.sign_widths"]]+ sv = np.array(g.metadata["prism.hadamard.sign_values"], dtype=np.float32)+ signs, at = {}, 0+ for w in sw:+ signs[w] = sv[at:at + w]+ at += w+ repo = hf_slice.Repo(a.repo)+ pre = "model.language_model.layers.%d.mlp.experts." % a.layer+ prof = {} # (who, role, expert) -> per-column mean squared error / mean+ for e in [int(x) for x in a.experts.split(",")]:+ gu = repo.tensor(pre + "gate_up_proj", e)[0]+ for role, W in (("gate", gu[:640]), ("up", gu[640:])):+ ti = by["blk.%d.ffn_%s_exps.weight" % (a.layer, role)]+ t = tqenc.TYPE_BY_NAME[ti.type_name]+ n_in, n_out = ti.shape[0], ti.shape[1]+ bb = tqenc.block_bytes(t)+ rb = n_in // 128 * bb+ raw = np.frombuffer(mm, dtype=np.uint8, count=n_out * rb, offset=g.data_start + ti.offset + e * n_out * rb)+ Wd = tqenc.decode(raw, t).reshape(n_out, n_in)+ Wr = tqenc.rotate(W, signs[n_in])+ packed, _, _, _ = tqenc.encode(Wr, t, iters=1, threads=8)+ Wo = tqenc.decode(packed, t).reshape(n_out, n_in)+ for who, Wq in (("theirs", Wd), ("ours", Wo)):+ c = ((Wr - Wq).astype(np.float64) ** 2).mean(axis=0)+ # remove the part explained by the signal's own column energy (weights of large columns err more)+ prof[(who, role, e)] = c / c.mean()+ print("expert %d %s fetched and encoded" % (e, role), flush=True)+ es = [int(x) for x in a.experts.split(",")]+ for who in ("theirs", "ours"):+ cc = [np.corrcoef(prof[(who, "gate", es[0])], prof[(who, "gate", e)])[0, 1] for e in es[1:]]+ cu = [np.corrcoef(prof[(who, "gate", e)], prof[(who, "up", e)])[0, 1] for e in es]+ print("%-7s corr of the per-column error profile: gate expert %d vs other experts' gate %s | gate vs up of the same expert %s" % (+ who, es[0], " ".join("%.3f" % x for x in cc), " ".join("%.3f" % x for x in cu)))+ return 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/tools/trellis/gyro_error_profile.py b/tools/trellis/gyro_error_profile.pyindex 3ead8b027..97c6b11b8 100644--- a/tools/trellis/gyro_error_profile.py+++ b/tools/trellis/gyro_error_profile.py@@ -31,7 +31,8 @@ def main() -> int:ap.add_argument("--roles", default="gate,down")a = ap.parse_args()g = gguf_reader.GGUFFile(a.gguf)- mm = mmap.mmap(open(a.gguf, "rb").fileno(), 0, access=mmap.ACCESS_READ)+ fh = open(a.gguf, "rb")+ mm = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ)by = {t.name: t for t in g.tensors}sw = [int(x) for x in g.metadata["prism.hadamard.sign_widths"]]sv = np.array(g.metadata["prism.hadamard.sign_values"], dtype=np.float32)@@ -56,7 +57,7 @@ def main() -> int:raw = np.frombuffer(mm, dtype=np.uint8, count=n_out * rb, offset=g.data_start + ti.offset + e * n_out * rb)Wd = tqenc.decode(raw, t).reshape(n_out, n_in)Wr = tqenc.rotate(W, signs[n_in])- packed, _, _, _ = tqenc.encode(Wr, t, iters=3, ncand=3)+ packed, _, _, _ = tqenc.encode(Wr, t, iters=3, ncand=3, threads=8)Wo = tqenc.decode(packed, t).reshape(n_out, n_in)Ed, Eo = (Wr - Wd).astype(np.float64), (Wr - Wo).astype(np.float64)print("L%d e%d %s %s" % (layer, e, role, ti.type_name))