Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
DFlash 2 speculative decoding — can PortOS run Qwen this way?
Date: 2026-08-19
Evaluation of DFlash 2 as a way to run the Qwen models faster across PortOS's local AI provider stack: the Ollama backend, the Claude-Ollama TUI provider, and the OpenCode-Ollama TUI provider.
Verdict
Not adoptable today, through any backend PortOS orchestrates. DFlash 2 is a
drafter checkpoint plus engine-side decoding machinery, and every engine that
can run it is either one PortOS does not orchestrate (vLLM, SGLang) or reachable
only from an unmerged pull request (llama.cpp). Ollama — the backend that
actually serves all three providers named above — ships DFlash v1 on its MLX
runner and has no DFlash 2 modules, no -dflash tag for the Qwen3.8 targets,
and an open credibility problem with the DFlash tag it does publish.
No PortOS feature is warranted. What the evaluation did surface was a live hazard the DFlash 2 release makes worse, which is fixed in the same change — see What shipped instead.
What DFlash 2 is
A speculative-decoding algorithm, not a quantization format, a sampling setting, or a model. A small drafter proposes a block of tokens and the target model verifies the whole block in one forward pass. DFlash 2 adds two modules on top of DFlash:
- Parallel path selection — keeps the top 16 candidates per position, scores adjacent pairs with a small bilinear attention, and traces one coherent path by dynamic programming (~2M extra parameters).
- Suffix decay correction — a two-tap dynamic depthwise convolution before and after each attention and feed-forward sublayer, so information crosses the block while positions still compute in parallel (~3% extra parameters).
Reported at 2.7–3.4× decode throughput on Qwen3.8-27B. Both modules live in the engine, which is why the drafter weights alone are not enough.
Why it cannot reach PortOS's stack
The TUI providers are not the integration point
The Claude-Ollama and OpenCode-Ollama providers are HTTP clients of the Ollama
daemon (isOllamaBackedProvider in server/lib/aiToolkit/internal/ollamaBacked.js
is the whole relationship — a base URL and a model list). Decoding strategy is
invisible to them: if the daemon decoded speculatively, both would get the
speedup with zero PortOS changes. So "implement DFlash 2 in the TUI providers"
is not a task that exists. The entire question is whether the daemon can do it.
Ollama — no
Verified against Ollama 0.32.14:
- Ollama has
x/models/dflash, documented as a "DFlash block-diffusion draft model: qwen3-shaped layers drafting a whole block per forward". That is DFlash v1. It has no candidate selector and no depthwise-convolution taps, so a DFlash 2 checkpoint's extra tensors have nowhere to load. ollama createdoes expose the drafter pairing (DRAFTdirective,--draft-quantize), but pairing is not the missing piece — the verify-side algorithm is.- The
qwen3.8:27btag list carries-mlx,-nvfp4,-mxfp8,-q4_K_M,-q8_0and-mtp-*variants. There is no-dflashtag for it at all. - The one DFlash tag Ollama does publish for another model was reported to decode at the same tokens/second as the non-DFlash build, and the issue was closed as not planned. Even the v1 path is not demonstrably live.
LM Studio — no
LM Studio's speculative decoding pairs a target with a drafter that shares its vocabulary, and its MTP support keys on models with a built-in MTP head. Neither path knows the DFlash 2 architecture.
llama.cpp, vLLM, SGLang — real, but out of reach
Support exists in all three and in none of them as a release:
| Engine | DFlash 2 status | Why PortOS can't use it |
|---|---|---|
| llama.cpp | PR #27342, open (2026-08-18) | Needs a from-source CMake build of an unmerged branch; PortOS orchestrates no llama.cpp server |
| vLLM | PR #52816, unmerged | Installed from a PR ref; Python serving stack PortOS does not manage |
| SGLang | PR #35371, unmerged | Same |
| oMLX | prebuilt DMG, GUI-configured | A separate third-party app, not a PortOS-managed backend |
Adopting any of them means PortOS taking on a third local inference backend — one built from an unmerged patch, on hardware assumptions PortOS does not make — to chase a speedup on one model family. That fails the project's own bar for adding surface area, and it is the same conclusion the Qwen3.8 27B MLX evaluation reached about MTP sidecars three days earlier: the complete, engine-native model is the correct integration boundary, and PortOS does not own a speculative pairing runtime.
When to re-evaluate
Re-open this when either holds, because either one removes all the work:
- Ollama publishes a packaged DFlash 2 tag for a Qwen target (a
qwen3.8:27b-dflash2-shaped tag that resolves to one self-contained model), and it measurably decodes faster thanqwen3.8:27b-mlxon the same box. - llama.cpp #27342 merges and Ollama's vendored engine picks it up.
Either way PortOS's work is a catalog entry, not a backend. Tracked in
#4568 (closed: resolved in PR #4570 with the OpenCode llama TUI provider and llama-server management).
What shipped instead
DFlash 2's release put a new hazard into PortOS's live Hugging Face model
search. Drafter checkpoints are non-standalone: incoai/Qwen3.8-27B-DFlash2 is
~2B of sidecar for a 27B target and produces nothing usable on its own. The
catalog already excluded MTP/drafter repos — but only on the MLX branch. The
GGUF branch had no drafter filter at all, and DFlash 2 ships GGUF drafters
(incoai/Qwen3.8-27B-DFlash2-GGUF, z-lab/…, and a growing long tail), so
searching a Qwen model in PortOS could offer one as a normal one-click Ollama or
LM Studio install. The user would get a model that cannot chat.
server/services/huggingFaceCatalog.js now filters both branches. The predicate
is deliberately narrow — the publisher's own draft-model / drafter tag:
Measured over one sample of 960 popular GGUF/MLX repos (the top downloads for twelve common model queries, both formats):
- An
mtporspeculative-decodingtag matches 68 of them (7.1%), the large majority being complete models that merely preserve a built-in MTP head (unsloth/Qwen3.6-27B-MTP-GGUF,huihui-ai/Huihui-Qwen3.6-27B-abliterated-MTP-GGUF). Filtering on those would hide mainstream installs — a worse failure than the one being fixed. - The
draft-model/draftertag matches 6 (0.6%), all of them genuine drafter sidecars.
The MLX branch keeps its pre-existing repo-name matching (that space is curated,
so a -MTP-/-DFlash- suffix is reliable there) and gains dflash\d* so an
MLX-published DFlash drafter is caught too.
Residual, accepted: a few GGUF drafters declare no drafter tag and no
distinguishing pipeline (Anbeeld/Qwen3.6-27B-DFlash-GGUF,
Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp). Nothing in their metadata
separates them from a complete model, and guessing from the name is what takes
out the mainstream repos above.
Sources
- DFlash 2 announcement
- DFlash paper · z-lab/dflash
incoai/Qwen3.8-27B-DFlash2· GGUF build- llama.cpp #27342 — spec: add DFlash2 support
- Ollama
x/models/dflash - Ollama #17683 — DFlash does not seem to be working
- LM Studio speculative decoding
- Qwen3.8 27B MLX options on macOS — the prior sidecar decision