Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.
Python agent-orchestrationaiai-agentschatopsdiscordlangchainmcpmicrosoft-teamsmulti-agentopen-sourceopenaiself-hostedslack
Document Strands Decider 4B as the default decision model
agent-runner35310066ec48ee9
diff --git a/decision/README.md b/decision/README.mdindex 86898f6af..c5ccf08b0 100644--- a/decision/README.md+++ b/decision/README.md@@ -9,6 +9,43 @@ the model's confidence clears a threshold.It is a small FastAPI app (`switch_decision`). It holds no state and keeps nomessage content after answering.+## The default model++Production runs **Strands Decider 4B** (`StrandsAgents/strands-decider-E4B-gemma4-v1-2610`,+served as `strands-decider-E4B`) on one NVIDIA GPU, through the `systemone`+backend. The Helm chart's default (`decision.modelServer.kind: strands`) runs+it in a sidecar from the `switch-decision-strands` image+(`model-servers/strands/`). Local compose keeps **Qwen3.5-4B** on llama.cpp as+its default, because that one runs on a CPU.++Measured on one NVIDIA L4 (AWS g6.xlarge, eu-west-3) over 100 routing+scenarios (`eval/scenarios_hard.json` and `eval/scenarios_extra60.json`, the+60 written blind to the results), threshold 0.6:++| Model | Correct | Wrong picks | Stayed quiet wrongly | p50 / p95 |+|---|---|---|---|---|+| Strands Decider 4B (bf16) | **92/100** | **2** | 6 | 0.27 / 0.30 s |+| Qwen3.5-4B Q4_K_M (llama.cpp) | 88/100 | 9 | 3 | 0.16 / 0.17 s |++The wrong picks decide it: an agent answering a message that was not for it+is the failure people notice, and most of Qwen's errors were that (a question+for a named person, people talking among themselves). Strands Decider's misses+sat at 0.51–0.61 confidence, just under the threshold.++- **Licence.** `strands-decider` is Apache-2.0, but the 4B checkpoint is built+ on Google's Gemma 4 (`google/gemma-4-E4B-it`), whose terms apply to whoever+ deploys it. Nothing is baked into the image: the weights download from+ Hugging Face at deploy time into the deployment's own volume, and Switch does+ not redistribute them.+- **Hardware.** A GPU with 24 GB (an L4), an NVIDIA driver for CUDA 13.0 (580+ or later), and 32 GiB of RAM on the host: the 16 GB of bf16 weights pass+ through memory while loading. With 16 GiB, start-up took about 10.5 minutes;+ the server's port opens only once the model is loaded.+- **Switching to Qwen:** `decision.modelServer.kind: llamacpp` in the chart.+ The service needs nothing else: the backend (`llamacpp`) and model name+ follow the kind. In compose, Qwen is the `decision` and `decision-gpu`+ profiles, Strands Decider the `decision-strands-gpu` profile.+## Contractswitch-core talks only to this contract; the backend behind it is configuration.@@ -49,8 +86,8 @@ switch-core talks only to this contract; the backend behind it is configuration.| `DECISION_BACKEND` | What it calls | Default `DECISION_MODEL` ||---|---|---|-| `llamacpp` (default) | a llama.cpp server; reads the probabilities of the option letters from the first token | `qwen3.5-4b-q4_k_m` |-| `systemone` | a SystemOne-API server (Clef-flash, Strands Decider) | `clef-flash` |+| `llamacpp` (the image's default) | a llama.cpp server; reads the probabilities of the option letters from the first token | `qwen3.5-4b-q4_k_m` |+| `systemone` | a SystemOne-API server: Strands Decider (the chart's default, `DECISION_MODEL=strands-decider-E4B`) or Clef-flash | `clef-flash` || `anthropic` | Claude Haiku on the Anthropic API, forced to call a pick tool; thinking off | `claude-haiku-5-5` || `bedrock` | a model on Amazon Bedrock through Converse, forced to call the same tool | `us.anthropic.claude-haiku-5-5-v1:0` || `ollama` | an Ollama model forced into JSON, kept for comparison | `qwen3.5:9b` |@@ -91,8 +128,30 @@ uv run pytest## Run with DockerThe `switch-decision` image is CPU-only and holds no model weights. The model-runs in its own container: on a GPU host, the llama.cpp server image with the-weights in a volume.+runs in its own container, with the weights in a volume.++Strands Decider (Linux, an NVIDIA GPU with 24 GB, the NVIDIA Container+Toolkit). `switch-decision-strands` serves on :8400; the first start+downloads the checkpoint and its base model (about 17 GB) into `/models`.+`HF_TOKEN` is optional; arguments after the image name are appended to+`strands-decider serve`:++```+docker build -t switch-decision-strands decision/model-servers/strands/+docker run -d --gpus all -v decision-hf:/models -p 8400:8400 switch-decision-strands+docker run -d -p 8230:8230 \+ --add-host=host.docker.internal:host-gateway \+ -e DECISION_BACKEND=systemone -e DECISION_MODEL=strands-decider-E4B \+ -e SYSTEMONE_URL=http://host.docker.internal:8400/v1/systemone \+ switch-decision+```++The image pins `strands-decider==0.2.0` and every dependency by hash+(`model-servers/strands/requirements.txt`, compiled from `requirements.in`+with the command at its top). `STRANDS_CHECKPOINT` and `STRANDS_MODEL_NAME`+choose another checkpoint.++Qwen3.5-4B on the llama.cpp server image:```docker build -t switch-decision decision/@@ -136,8 +195,10 @@ accuracy and the median and maximum latency.On the 40-scenario hard set:-- **Qwen3.5-4B (4-bit) via llama.cpp:** 34/40, 0.61 s median on an Apple M4 Pro.+- **On one NVIDIA L4:** Strands Decider 4B 36/40 at 0.20 s median, Qwen3.5-4B+ (4-bit) 34/40 at 0.15 s, Strands Decider 2B 29/40 at 0.11 s. The+ 100-scenario comparison is under "The default model" above.- **Haiku 5.5 via the Anthropic API:** 39/40, 1.4–1.6 s median, about 2.7 sp95. About 0.65 s of each call is fixed API and network overhead.-- **On one NVIDIA L4:** an estimated 0.1–0.4 s for the small models. Not yet- measured.+- **On an Apple M4 Pro (no NVIDIA GPU):** Qwen3.5-4B 34/40 at 0.61 s median;+ Strands Decider 4B 36/40 at 1.06 s, and it needs 16 GB of the Mac's memory.diff --git a/docs/old/LOCAL_DEVELOPMENT.md b/docs/old/LOCAL_DEVELOPMENT.mdindex 98010ee2c..f791e69a2 100644--- a/docs/old/LOCAL_DEVELOPMENT.md+++ b/docs/old/LOCAL_DEVELOPMENT.md@@ -234,6 +234,17 @@ llama.cpp server for its model, Qwen3.5-4B in 4-bit, whose weights (about`decision` profile; `just decision-up gpu` is the `decision-gpu` profile, forLinux with an NVIDIA GPU and the NVIDIA Container Toolkit.+Production (the Helm chart) defaults to a different model, **Strands Decider+4B**, which made far fewer wrong picks on our routing scenarios (2 against+Qwen's 9 in 100) but needs an NVIDIA GPU with 24 GB, so it can't be the local+default. `just decision-up strands` (the `decision-strands-gpu` profile) runs+it on Linux with an NVIDIA GPU (24 GB), about 24 GiB of free RAM and the+NVIDIA Container Toolkit; there is no GPU passthrough on macOS. It builds the+`switch-decision-strands` image, and the first start downloads the checkpoint+and its Gemma 4 base model (about 17 GB, Gemma terms apply) into the+`decision-hf` volume, then loads it, which takes minutes; `HF_TOKEN` is+optional. `decision/README.md` has the numbers and the licence note.+switch-core runs on the host, so point it at the service in `.env` andrestart `just run`:@@ -253,7 +264,7 @@ out, not for judging its latency. Another model is `DECISION_MODEL_URL`,| Command | What it does || --- | --- |-| `just decision-up [gpu]` / `just decision-down` | Start / stop the decision service for auto-route (see above) |+| `just decision-up [gpu\|strands]` / `just decision-down` | Start / stop the decision service for auto-route (see above) || `just local-cloud` / `just local-cloud-console` | Run switch-core as a stand-in for Switch Cloud, and the Console pointed at it (see above) || `just format` / `just check` | Format with ruff / lint-check in CI mode (no changes) || `just typecheck` | mypy over `core/switch_core/` |diff --git a/docs/old/decision-routing.md b/docs/old/decision-routing.mdindex b2d198256..da78edd25 100644--- a/docs/old/decision-routing.md+++ b/docs/old/decision-routing.md@@ -32,6 +32,39 @@ With no `DECISION_ENDPOINT_URL`, there is no server default: auto-route isunavailable in every workspace until its admins choose a backend, and Haiku isnot offered (it runs in the server's decision service).+### The model behind the server default++The bundled Helm chart runs **Strands Decider 4B** (AWS's Strands Decider,+the checkpoint built on Google's Gemma 4 E4B) on one NVIDIA GPU, beside the+decision service. It was chosen on a benchmark of 100 routing scenarios in+four rooms with overlapping agents, a third of them messages nobody should+answer, run on one NVIDIA L4:++| Model | Correct | Wrong picks | Stayed quiet when it shouldn't have | Median |+| --- | --- | --- | --- | --- |+| Strands Decider 4B | 92 | 2 | 6 | 0.27 s |+| Qwen3.5-4B, 4-bit (llama.cpp) | 88 | 9 | 3 | 0.16 s |+| Claude Haiku 5.5 (40-scenario subset, for scale) | 38/40 | 0 | 2 | 1.5 s |++The four-point gap is within noise at 100 scenarios; the wrong picks decide+it. A wrong pick is the failure people notice, an agent jumping into a+conversation that was not for it, and most of Qwen's were exactly that+(answering a message meant for a named person, or people talking among+themselves). Strands Decider's misses are the quiet kind: it stays out, its+confidence usually just under the 0.6 threshold.++- **Switching to Qwen** is one chart value, `decision.modelServer.kind:+ llamacpp`, which runs Qwen3.5-4B on llama.cpp's CUDA server instead (smaller:+ 8 GiB of RAM rather than 24). Local compose uses Qwen by default, because it+ runs on a CPU; see `docs/old/LOCAL_DEVELOPMENT.md`.+- **Licence.** Strands Decider is Apache-2.0, but its 4B checkpoint is built on+ Google's Gemma 4 (`google/gemma-4-E4B-it`), whose terms apply to whoever+ deploys it. The weights download from Hugging Face at deploy time, into the+ deployment's own volume; Switch does not redistribute them.+- **Hardware.** One NVIDIA GPU with 24 GB (an L4) and a node with 32 GiB of+ RAM (an AWS g6.2xlarge): the 16 GB of bf16 weights pass through memory while+ they load. With 16 GiB the model took about 10 minutes to start.+## Each workspace's choiceA workspace's admins (its owners and admins, and the server operator) choose