Skip to content

Qwen3.8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

C++

trellis: README, column-profile check of the Gyro error

Niko12216f99190f8260
diff --git a/tools/trellis/README.md b/tools/trellis/README.md
new file mode 100644
index 000000000..a76354aa7
--- /dev/null
+++ b/tools/trellis/README.md
@@ -0,0 +1,34 @@
+# Trellis encoder (research, branch f-dsv41-enc)
+
+An encoder for the trellis expert formats that Strata already decodes (TQ2_T, TQK6, TQK7 of agentionai's llama.cpp fork)
+and for two experimental lower rates, TQK5X (22 bytes per 128 weights, 1.375 bpw) and TQK4X (18 bytes, 1.125 bpw).
+Written for the DeepSeek-V4.1-Flash compression plan (gate 2 and the start of gate 3). Nothing here changes an existing
+type, layout or default; the engine's CPU decoder only gains K = 5 and 4 (ids 224 / 225, experimental version 1).
+
+| file | what |
+| --- | --- |
+| `tqenc.hpp`, `tqenc.cpp` | the encoder: tail-biting Viterbi on the 2^15-state shift trellis, fp16 scale refinement, importance weights; a C API |
+| `tqenc_test.cpp` | the tests (about 64,000 checks): streams and the circular wrap, Viterbi against a naive DP, planted solutions, bit-exact decode against `tq_states.hpp` and the scalar / AVX2 / AVX-512 kernels, special blocks, thread determinism, quality against the rate-distortion limit |
+| `tqenc_hip.cpp`, `tqenc_hip_test.cpp` | the same search on a GPU (HIP, gfx1201): bit-identical blocks, about 20 M weights/s on one R9700 |
+| `../tqenc.py` | Python wrapper (ctypes): `encode`, `decode`, the input-only block-128 Hadamard, signs, the `.tqx` container |
+| `pilot.py`, `pilot_summary.py` | the block-quality pilot on a source GGUF (local files or HTTP range requests) |
+| `gyro_reencode.py`, `gyro_vs_original.py`, `gyro_error_profile.py`, `hf_slice.py` | checks against agentionai's Gyro files and the original Qwen weights |
+
+Build and test (Linux x86-64, g++ with AVX2; the GPU part needs ROCm):
+
+ sh tools/trellis/build.sh test && build-tqenc/tqenc_test # --quick skips the quality table
+ sh tools/trellis/build.sh lib && python3 tools/trellis/test_tqenc_py.py
+ sh tools/trellis/build.sh hip && HIP_VISIBLE_DEVICES=3 build-tqenc/tqenc_hip_test
+
+`build.sh hip` compiles `tqenc.cpp` with g++ on purpose: under hipcc, `ggml-common-tq.h` turns its codebook into a
+`__device__` variable and the host side reads zeros.
+
+## Format facts the encoder relies on (checked against the decoder, not the comments)
+
+* A block is `{fp16 d; 4K bytes}` for 128 weights: 32 steps, step t gives weights 4t..4t+3.
+* State of step t: 15 bits. TQK: the 16-bit window of the stream at bit `(31 - t) K` (LSB first, circular modulo 32 K bits), masked
+ to 15 bits. TQ2_T: `(qs[(t+31)%32] << 8 | qs[t]) & 0x7fff`.
+* Weights of a step: `x = state * 0x9e3779b1`, the two codebook points `lut[x >> 21]` and `lut[(x >> 10) & 2047]`, times d. The
+ decoder's table is `round(fp16 * 8192)`; the encoder uses that table, so its reconstruction is the decoder's, bit for bit.
+* A legal stream is any byte string (every state is a window of the same circular stream); tail-biting only matters to the search.
+* K5 and K4 are the TQK rule with K = 5 and 4. Nothing else changes (hash, codebook, mask, bit order, scale).
diff --git a/tools/trellis/gyro_column_profile.py b/tools/trellis/gyro_column_profile.py
new file mode 100644
index 000000000..cd212dc27
--- /dev/null
+++ b/tools/trellis/gyro_column_profile.py
@@ -0,0 +1,70 @@
+"""gyro_column_profile.py - is the unevenness of agentionai's per-input-column error a property of the layer (an
+activation-aware objective shared by all experts that read the same input) or noise? Compares the per-column squared-error
+profile (rotated basis) between experts and between gate and up of one layer, for their file and for our plain encode.
+"""
+from __future__ import annotations
+
+import argparse
+import mmap
+import pathlib
+import sys
+
+import numpy as np
+
+HERE = pathlib.Path(__file__).resolve().parent
+sys.path.insert(0, str(HERE.parent))
+sys.path.insert(0, str(HERE))
+import gguf_reader # noqa: E402
+import hf_slice # noqa: E402
+import tqenc # noqa: E402
+
+
+def main() -> int:
+ ap = argparse.ArgumentParser()
+ ap.add_argument("--gguf", required=True)
+ ap.add_argument("--repo", default="Qwen/Qwen3.8-Flash-Next")
+ ap.add_argument("--layer", type=int, default=10)
+ ap.add_argument("--experts", default="0,1,2,3")
+ a = ap.parse_args()
+ g = gguf_reader.GGUFFile(a.gguf)
+ fh = open(a.gguf, "rb")
+ mm = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ)
+ by = {t.name: t for t in g.tensors}
+ sw = [int(x) for x in g.metadata["prism.hadamard.sign_widths"]]
+ sv = np.array(g.metadata["prism.hadamard.sign_values"], dtype=np.float32)
+ signs, at = {}, 0
+ for w in sw:
+ signs[w] = sv[at:at + w]
+ at += w
+ repo = hf_slice.Repo(a.repo)
+ pre = "model.language_model.layers.%d.mlp.experts." % a.layer
+ prof = {} # (who, role, expert) -> per-column mean squared error / mean
+ for e in [int(x) for x in a.experts.split(",")]:
+ gu = repo.tensor(pre + "gate_up_proj", e)[0]
+ for role, W in (("gate", gu[:640]), ("up", gu[640:])):
+ ti = by["blk.%d.ffn_%s_exps.weight" % (a.layer, role)]
+ t = tqenc.TYPE_BY_NAME[ti.type_name]
+ n_in, n_out = ti.shape[0], ti.shape[1]
+ bb = tqenc.block_bytes(t)
+ rb = n_in // 128 * bb
+ raw = np.frombuffer(mm, dtype=np.uint8, count=n_out * rb, offset=g.data_start + ti.offset + e * n_out * rb)
+ Wd = tqenc.decode(raw, t).reshape(n_out, n_in)
+ Wr = tqenc.rotate(W, signs[n_in])
+ packed, _, _, _ = tqenc.encode(Wr, t, iters=1, threads=8)
+ Wo = tqenc.decode(packed, t).reshape(n_out, n_in)
+ for who, Wq in (("theirs", Wd), ("ours", Wo)):
+ c = ((Wr - Wq).astype(np.float64) ** 2).mean(axis=0)
+ # remove the part explained by the signal's own column energy (weights of large columns err more)
+ prof[(who, role, e)] = c / c.mean()
+ print("expert %d %s fetched and encoded" % (e, role), flush=True)
+ es = [int(x) for x in a.experts.split(",")]
+ for who in ("theirs", "ours"):
+ cc = [np.corrcoef(prof[(who, "gate", es[0])], prof[(who, "gate", e)])[0, 1] for e in es[1:]]
+ cu = [np.corrcoef(prof[(who, "gate", e)], prof[(who, "up", e)])[0, 1] for e in es]
+ print("%-7s corr of the per-column error profile: gate expert %d vs other experts' gate %s | gate vs up of the same expert %s" % (
+ who, es[0], " ".join("%.3f" % x for x in cc), " ".join("%.3f" % x for x in cu)))
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/tools/trellis/gyro_error_profile.py b/tools/trellis/gyro_error_profile.py
index 3ead8b027..97c6b11b8 100644
--- a/tools/trellis/gyro_error_profile.py
+++ b/tools/trellis/gyro_error_profile.py
@@ -31,7 +31,8 @@ def main() -> int:
ap.add_argument("--roles", default="gate,down")
a = ap.parse_args()
g = gguf_reader.GGUFFile(a.gguf)
- mm = mmap.mmap(open(a.gguf, "rb").fileno(), 0, access=mmap.ACCESS_READ)
+ fh = open(a.gguf, "rb")
+ mm = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ)
by = {t.name: t for t in g.tensors}
sw = [int(x) for x in g.metadata["prism.hadamard.sign_widths"]]
sv = np.array(g.metadata["prism.hadamard.sign_values"], dtype=np.float32)
@@ -56,7 +57,7 @@ def main() -> int:
raw = np.frombuffer(mm, dtype=np.uint8, count=n_out * rb, offset=g.data_start + ti.offset + e * n_out * rb)
Wd = tqenc.decode(raw, t).reshape(n_out, n_in)
Wr = tqenc.rotate(W, signs[n_in])
- packed, _, _, _ = tqenc.encode(Wr, t, iters=3, ncand=3)
+ packed, _, _, _ = tqenc.encode(Wr, t, iters=3, ncand=3, threads=8)
Wo = tqenc.decode(packed, t).reshape(n_out, n_in)
Ed, Eo = (Wr - Wd).astype(np.float64), (Wr - Wo).astype(np.float64)
print("L%d e%d %s %s" % (layer, e, role, ti.type_name))