A high-performance inference engine for AI models
Inference benchmarks
Benchmark adapters for inference engines. Each runner loads one model, reads requests from standard input, and writes generated text, timing, throughput, and process memory measurements as JSON Lines. A process can handle multiple requests without reloading its model.
Interaction with every engine follows the same loop: after launch, it waits for a request on stdin, runs inference, writes the result to stdout, and then waits for the next request.
Each engine includes adapter source files that collect the same set of benchmark metrics. Memory usage is measured using the shared API in memory_counters.h.
Each engine supports:
- Argument
-mor--modelwith value HuggingFace repository id or local model path. - One shot execution:
for example${RUN_COMMAND} -m "${MODEL}" <<'EOF' ${INPUT_JSON} EOF./engine-llamacpp/run.sh -m "unsloth/Qwen3.5-0.8B-GGUF:Q4_K_M" <<'EOF' {"prompt_text": "Tell me about London"} EOF
When max_tokens is omitted, null, or 0, decoding continues until an EOS token.
Engines
llama.cpp
./engine-llamacpp/run.sh -m "unsloth/Qwen3.5-0.8B-GGUF:Q4_K_M"
Also DFlash is supported
./engine-llamacpp/run.sh -m "unsloth/Qwen3.6-27B-GGUF:Q4_K_S" \
-d "ggml-org/Qwen3.6-27B-GGUF:BF16"
ML Drift
./engine-mldrift/run.sh -m "Qwen/Qwen3-0.6B"
Requires bazelisk (brew install bazelisk) and, for Hugging Face ids, the hf CLI. The Metal LLM sample runner supports the gemma3:270m, gemma3:1b, gemma4:12b, qwen3:0.6b, qwen3:1.7b, qwen3:8b, and qwen3:14b presets (the last added by the benchmark patch), selected from the checkpoint's config.json. The first run of a checkpoint converts its float weights with upstream extract_weights_hf through Bazel: to Q4_0 (block size 32) by default, or unquantized with --precision f16.
Prompts are tokenized by ML Drift's own tokenizer. Chats use its turn markers for user, assistant and system turns (Gemma 3 has no system turn); tools, tool calls and reasoning content are rejected. Decoding is greedy only, so requests with sampling or a positive speculative_depth are rejected. The prefill graph is compiled for each prompt length, so each request rebuilds the runner before its first run.
Compare against uzu on the same Qwen3 checkpoint at 4 bits (uzu runs a lalamo conversion of mlx-community/<model>-4bit, ML Drift its Q4_0 extraction), or at 16 bits with --bits 16:
uv run compare/uzu_mldrift.py --model Qwen/Qwen3-14B --bits 4 --num-runs 5 --max-tokens 256
MLX
uv run --project engine-mlx bench-mlx -m "mlx-community/Qwen3.5-0.8B-4bit"
mlx-serve
Requires Zig 0.17, pkg-config and libwebp.
./engine-mlxserve/run.sh -m "mlx-community/Qwen3.5-0.8B-4bit"
Supported native MTP heads are detected and loaded automatically from the target checkpoint, including sidecars. A detected head that fails to load or bind produces an error. Models without a supported head use ordinary autoregressive decoding.
Pass a compatible DFlash or Gemma assistant checkpoint with --draft-model (or -d) to use external speculative decoding. Both paths accept local directories or Hugging Face repositories; an explicit draft model takes precedence over MTP.
./engine-mlxserve/run.sh -m "mlx-community/Qwen3.6-27B-4bit" \
--draft-model "z-lab/Qwen3.6-27B-DFlash"
Omit speculative_depth in the JSON request for the decoder's default depth, set it to 0 for ordinary decoding, or supply a positive depth within the decoder's supported range. DFlash uses upstream's hardware-dependent default block size. Depth counts proposed tokens; a DFlash verification block contains one additional target token. Upstream's adaptive acceptance gates can reduce speculation during a request. Inference runs directly through Zig sources, without HTTP.
tokens_per_forward_pass counts emitted tokens per target forward, including prefill, verification, and any replay. Draft-head forwards are excluded. Each run starts with fresh generation caches and adaptive timing/acceptance history.
MTPLX
uv run --project engine-mtplx bench-mtplx -m "Youssofal/Qwen3.5-4B-MTPLX-Optimized-Speed"
MTPLX uses the model's multi-token prediction (MTP) heads to draft tokens and verifies them with the target model: this is speculative decoding without a separate draft model. Omit speculative_depth to use the model pack's recommended depth (fallback: 3), set it to 0 for ordinary autoregressive decoding, or supply a positive depth to override it. Positive depths require matching MTP weights.
oMLX
uv run --project engine-omlx bench-omlx -m "mlx-community/SmolLM-135M-Instruct-4bit"
Native MTP is enabled automatically when the target checkpoint has embedded prediction heads supported by oMLX's text engine.
uv run --project engine-omlx bench-omlx -m "TheWirelessPhoenix/Qwen3.5-4B-oQ4e-mtp"
Also DFlash is supported
uv run --project engine-omlx bench-omlx -m "mlx-community/Qwen3.6-27B-4bit" \
-d "z-lab/Qwen3.6-27B-DFlash"
splash
uv run --project engine-splash bench-splash -m "unsloth/Qwen3.8-27B-GGUF:UD-IQ1_S"
DFlash already supported with hardcoded speculative_depth == 7
uzu
cargo run --release -p benchmarks-uzu -- -m "trymirai/LFM2.5-230M-M"
Current configuration enables DFlash only on M5, M5 Pro, and M5 Max
cargo run --release -p benchmarks-uzu -- -m "trymirai/Qwen3.5-9B-M"
iOS
platform-ios runs these engines inside the iOS app on an iPhone or iPad:
uv run --project platform-ios bench-ios run -e llamacpp -m "unsloth/Qwen3.5-0.8B-GGUF:Q4_K_M"
Setup, device selection, model downloads and pushes are in the app's README. pytest engines leaves it out because it needs a device; test it with uv run pytest --engine ios --ios-engine ENGINE --ios-model MODEL, adding --ios-device UDID or --ios-host HOST as needed.
Server
./server contains tool for easy benchmarking OpenAI-compatible server
uv run bench-server uzu \
--model "trymirai/Qwen3.5-4B-M" \
--input server/tasks/ai-history-summary.json \
--num-runs 3
Output will show average metrics.
By default, each run waits until CPU and GPU temperatures are at or below 60°C.
Add --no-wait-cooldown to send requests without waiting for cooling.
TensorFold requires Zig 0.17.
Tests
Unit tests
uv run pytest
All engines integration tests
uv run pytest engines
Specific engine test
uv run pytest --engine uzu