Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

docs(local-models) [DO NOT MERGE]: add vllm hidden-state serving notes (#13)

sabhatinas832547f0c2e1
diff --git a/docs/vllm-serve-hidden-state.md b/docs/vllm-serve-hidden-state.md
new file mode 100644
index 000000000..2e2a81123
--- /dev/null
+++ b/docs/vllm-serve-hidden-state.md
@@ -0,0 +1,106 @@
+# vLLM Hidden-State Serving for Local Models
+
+This page captures operational notes for serving a local vLLM model with hidden-state extraction enabled. The hidden-state connector writes prefill activations to `.safetensors` files and returns the actual file path in `kv_transfer_params.hidden_states_path`.
+
+Docker is not required by the protocol. Use Docker when you want a reproducible CUDA/vLLM runtime; use `vllm serve` directly when the local Python environment has a vLLM build that includes `extract_hidden_states` and `ExampleHiddenStatesConnector`.
+
+## Docker launch
+
+Pick one filesystem path for hidden states and mount it into the container. The container path used in `shared_storage_path` must be the same path clients pass as `kv_transfer_params.hidden_states_path`.
+
+```bash
+export HIDDEN_STATES_DIR=/tmp/vllm-hidden-states
+export HF_CACHE_DIR=/tmp/vllm-hf-cache
+mkdir -p "${HIDDEN_STATES_DIR}" "${HF_CACHE_DIR}"
+
+docker run -d --name vllm_qwen35 \
+ --gpus all \
+ --ipc=host \
+ -p 0.0.0.0:8000:8000 \
+ -v "${HF_CACHE_DIR}:/root/.cache/huggingface" \
+ -v "${HIDDEN_STATES_DIR}:${HIDDEN_STATES_DIR}" \
+ vllm/vllm-openai:latest-cu129 \
+ Qwen/Qwen3.6-35B-A3B \
+ --tensor-parallel-size 8 \
+ --max-model-len 32768 \
+ --reasoning-parser qwen3 \
+ --enable-auto-tool-choice \
+ --tool-call-parser hermes \
+ --no-enable-chunked-prefill \
+ --speculative-config '{"method":"extract_hidden_states","num_speculative_tokens":1,"draft_model_config":{"hf_config":{"eagle_aux_hidden_state_layer_ids":[39]}}}' \
+ --kv-transfer-config '{"kv_connector":"ExampleHiddenStatesConnector","kv_role":"kv_producer","kv_connector_extra_config":{"shared_storage_path":"/tmp/vllm-hidden-states"}}'
+```
+
+Use host IPC for long-running Docker jobs. The default Docker IPC mode gives the
+container a private 64 MiB `/dev/shm`, which can starve vLLM's tensor-parallel
+shared-memory broadcast path while hidden-state extraction is enabled.
+
+For `Qwen/Qwen3.6-35B-A3B`, layer `39` is the last hidden-state layer. To capture multiple layers, add each layer id to `eagle_aux_hidden_state_layer_ids`, for example `[0,1,2,39]`. Capturing all layers can make each probe much larger and may require a lower `--max-model-len` to leave enough KV-cache memory.
+
+## Direct vLLM CLI launch
+
+The direct CLI form serves the same model without Docker. There is no volume mount; `shared_storage_path` is a host path and clients must be able to read that same path.
+
+```bash
+export HIDDEN_STATES_DIR=/tmp/vllm-hidden-states
+mkdir -p "${HIDDEN_STATES_DIR}"
+
+vllm serve Qwen/Qwen3.6-35B-A3B \
+ --host 0.0.0.0 \
+ --port 8000 \
+ --tensor-parallel-size 8 \
+ --max-model-len 32768 \
+ --reasoning-parser qwen3 \
+ --enable-auto-tool-choice \
+ --tool-call-parser hermes \
+ --no-enable-chunked-prefill \
+ --speculative-config '{"method":"extract_hidden_states","num_speculative_tokens":1,"draft_model_config":{"hf_config":{"eagle_aux_hidden_state_layer_ids":[39]}}}' \
+ --kv-transfer-config '{"kv_connector":"ExampleHiddenStatesConnector","kv_role":"kv_producer","kv_connector_extra_config":{"shared_storage_path":"/tmp/vllm-hidden-states"}}'
+```
+
+Use the direct CLI only after confirming your installed vLLM accepts both `--speculative-config '{"method":"extract_hidden_states",...}'` and `--kv-transfer-config '{"kv_connector":"ExampleHiddenStatesConnector",...}'`. If those flags fail, use the known container image or install a vLLM build that contains the connector.
+
+## Verify one hidden-state file
+
+Send one Chat Completions request with `max_tokens=1`. The probe should return a `kv_transfer_params.hidden_states_path` value that points at a `.safetensors` file.
+
+```bash
+curl http://localhost:8000/v1/chat/completions \
+ -H "Content-Type: application/json" \
+ -d '{
+ "model": "Qwen/Qwen3.6-35B-A3B",
+ "messages": [{"role": "user", "content": "Return one short sentence."}],
+ "max_tokens": 1,
+ "kv_transfer_params": {
+ "hidden_states_path": "/tmp/vllm-hidden-states",
+ "include_output_tokens": false
+ }
+ }'
+```
+
+Read the path from the response rather than assuming a filename. vLLM may choose the concrete safetensors file name.
+
+```bash
+uv run python - <<'PY_INNER'
+from pathlib import Path
+from safetensors import safe_open
+
+path = Path("/tmp/vllm-hidden-states")
+files = sorted(path.glob("*.safetensors"), key=lambda item: item.stat().st_mtime)
+if not files:
+ raise SystemExit("no safetensors files written")
+
+with safe_open(files[-1], framework="numpy") as handle:
+ for key in handle.keys():
+ tensor = handle.get_tensor(key)
+ print(files[-1], key, tensor.shape, tensor.dtype)
+PY_INNER
+```
+
+## Troubleshooting
+
+- `probe response missing kv_transfer_params`: the server is not running with `ExampleHiddenStatesConnector`, or the request did not include `kv_transfer_params`.
+- `no safetensors files written`: check that `shared_storage_path` exists and is writable by the vLLM process.
+- `No available shared memory broadcast block found` followed by `RPC call to sample_tokens timed out`: relaunch the Docker container with `--ipc=host`.
+- Context-length startup errors from vLLM: lower `--max-model-len`, reduce the number of captured layers, or increase available GPU memory.
+- Hidden-state extraction does not work with chunked prefill; keep `--no-enable-chunked-prefill` in the launch command.
diff --git a/mkdocs.yml b/mkdocs.yml
index 537c1af6a..0409a52e7 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -41,7 +41,7 @@ nav:
exclude_docs: |
internal/**
research/**
- local_models.md
+ vllm-serve-hidden-state.md
theme:
name: material