Skip to content

FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.

docs: refine README.md, add quick start guide

jason-fxzfca27580a14e
diff --git a/README.md b/README.md
index b7d79390..f053311a 100644
--- a/README.md
+++ b/README.md
@@ -10,26 +10,30 @@
| <a href="https://www.flashml.ai/"><b>Download</b></a> | <a href="https://arxiv.org/abs/2608.16157"><b>Paper</b></a> | <a href="https://join.slack.com/t/flashml/shared_invite/zt-3zpdh5j10-9dwTXrgLiqpVxizhA9KVbA"><b>Developer Slack</b></a> | <a href="https://discord.gg/xzwSnMdsX"><b>Community Discord</b></a> |
</p>
-A local, MoE-offload inference runtime with an OpenAI- and Anthropic-compatible
-HTTP API — Run DeepSeek-V4-Flash on your 5090 with 20+ TPS.
+Unlock datacenter-class intelligence on the hardware you already own — Run 290B+ frontier MoE models locally on your gaming PC at blistering interactive speeds.
+## About
-## Quick start
+FreeToken is an edge-native Mixture-of-Experts (MoE) serving engine designed for running frontier-scale open-weight models on personal and consumer hardware. It treats heterogeneous edge resources—GPUs, CPUs, host memory, and interconnects—as a unified, elastic inference platform. Its core features include:
-See [docs/install.md](docs/install.md) for requirements and installation.
+- **Fast Edge-Native Runtime**: Provides efficient MoE serving with bandwidth-adaptive CPU–GPU co-execution ($q^\star$ policy), full-layer double-buffered prefill streaming, global LRU expert caching, graph-compatible execution, and the FTW fast weight format.
+- **Semantic-Aware Caching**: Features semantic anchor checkpoints for recurrent state and KV caches, allowing agentic context edits (e.g., tool calls, thinking blocks) to avoid redundant context recomputation.
+- **Elastic Memory Management**: Supports dynamic, runtime VRAM re-allocation between expert caches and KV memory without engine restarts or weight reloading.
+- **Broad MoE & Ecosystem Support**: Supports frontier open-weight MoE models (e.g., DeepSeek-V4-Flash, Qwen3.6-35B-A3B, GLM-5.2) across various parameter scales and quantization formats (e.g., MXFP4, NVFP4, FP8, BF16), with Anthropic/OpenAI-compatible APIs for seamless integration with real-world coding and tool-calling agents (e.g., Codex, Claude Code, OpenCode, OpenClaw, DeepSeek Harness).
+- **Diverse Consumer Hardware**: Scales across consumer laptops, gaming desktops, and workstation GPUs, with native support for NVIDIA RTX 30, RTX 40, and RTX 50 series GPUs.
-```bash
-ft serve --model ~/models/Qwen3.6-35B-A3B # API server on http://127.0.0.1:1919
-ft launch claude # point an agent at it (codex / dsh / hermes / opencode / openclaw)
-ft shell # or chat in the terminal
-```
+## Getting Started
-## Documentation
+**Desktop app** — download FreeToken for Windows or Linux at
+[flashml.ai](https://www.flashml.ai/). It sets the engine up for you and gives you a GUI for running models, chatting, and tuning the engine.
-- [Install](docs/install.md) — requirements and setup
-- [Supported models](docs/models.md) — model × quantization
-- [CLI reference](docs/cli.md) — `ft` commands and environment variables
+**CLI**:
+
+- [Install FreeToken](docs/install.md)
+- [Quick start](docs/quickstart.md)
+- [Supported models](docs/models.md)
+- [CLI reference](docs/cli.md)
## Acknowledgment
diff --git a/docs/install.md b/docs/install.md
index 38d353b7..f5205ab3 100644
--- a/docs/install.md
+++ b/docs/install.md
@@ -3,22 +3,23 @@
## Requirements
- Linux x86_64, NVIDIA GPU, driver r580+ (CUDA 13)
-- CUDA 13 toolkit with `nvcc` — compiles the C++ extensions at install time and
- JIT-compiles CUDA kernels on first use
-- Python >= 3.10 and [`uv`](https://docs.astral.sh/uv/)
+- Python >= 3.10, with [uv](https://docs.astral.sh/uv/) recommended (plain
+ `pip` + `venv` works too)
+
+## Method 1: Install from PyPI
```bash
-# install uv
-curl -LsSf https://astral.sh/uv/install.sh | sh
+uv venv && source .venv/bin/activate
+uv pip install "freetoken[accel]"
```
-## Install
+CUDA kernels are JIT-compiled on first use, need a CUDA 13 toolkit with `nvcc` on PATH.
+
+## Method 2: Install from source
```bash
git clone https://github.com/FlashML-org/FreeToken.git && cd FreeToken
-uv venv # create a virtual environment
-uv pip install -e . # install FreeToken in editable mode
-# if you need flashinfer/sglang-kernels, install the accel extras:
+uv venv && source .venv/bin/activate
uv pip install -e ".[accel]"
```
@@ -26,7 +27,10 @@ uv pip install -e ".[accel]"
```bash
source .venv/bin/activate
+ft --version
ft serve --model ~/path/to/Qwen3.6-35B-A3B
curl http://127.0.0.1:1919/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"Qwen3.6-35B-A3B","messages":[{"role":"user","content":"hi"}]}'
```
+
+Then head to [quickstart.md](quickstart.md).
diff --git a/docs/quickstart.md b/docs/quickstart.md
new file mode 100644
index 00000000..bea8d2e5
--- /dev/null
+++ b/docs/quickstart.md
@@ -0,0 +1,61 @@
+# Quick start
+
+Assumes FreeToken is installed — see [install.md](install.md).
+
+## Launch a server
+
+```bash
+ft serve --model ~/models/Qwen3.6-35B-A3B
+```
+
+`--model` also takes a Hugging Face repo id. Everything else — dtype, attention
+and MoE backends, cache sizes, tool-call and reasoning parsers — resolves from
+the checkpoint and the GPU; see [cli.md](cli.md) for the flags. The server is
+ready when the log reaches `API server is ready to serve on 127.0.0.1:1919`.
+
+## Send a request
+
+Check what is being served:
+
+```bash
+curl http://127.0.0.1:1919/v1/models
+```
+
+Then use that id as the `model` field:
+
+```bash
+curl http://127.0.0.1:1919/v1/chat/completions \
+ -H 'Content-Type: application/json' \
+ -d '{
+ "model": "Qwen3.6-35B-A3B",
+ "messages": [{"role": "user", "content": "What is a Mixture-of-Experts model?"}],
+ "max_tokens": 256,
+ "stream": true
+ }'
+```
+
+FreeToken serves the OpenAI API (`/v1/chat/completions`, `/v1/responses`,
+`/v1/models`) and the Anthropic API (`/v1/messages`,
+`/v1/messages/count_tokens`), so a client library for either works by pointing
+its base URL at the server.
+
+## Chat in the terminal
+
+A simple TUI to interact with the server:
+
+```bash
+ft shell # attach to the server above
+ft shell --model ~/models/Qwen3.6-35B-A3B # start an engine and chat, one process
+```
+
+`/help` lists the in-shell commands. Attach mode needs no GPU, so it also drives
+a server on another machine (`--server URL`).
+
+## Use a coding agent
+
+```bash
+ft launch claude # claude / codex / dsh / hermes / openclaw / opencode
+```
+
+Writes that agent's provider config, installs its CLI if missing, and starts it
+against your server. `--dry-run` previews the changes.