Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.
Python agent-orchestrationaiai-agentschatopsdiscordlangchainmcpmicrosoft-teamsmulti-agentopen-sourceopenaiself-hostedslack
Serve Strands Decider with its adapter merged into the base weights
agent-runner3538f2469737bf8
diff --git a/decision/README.md b/decision/README.mdindex c5ccf08b0..53b6f0dcf 100644--- a/decision/README.md+++ b/decision/README.md@@ -133,8 +133,12 @@ runs in its own container, with the weights in a volume.Strands Decider (Linux, an NVIDIA GPU with 24 GB, the NVIDIA ContainerToolkit). `switch-decision-strands` serves on :8400; the first startdownloads the checkpoint and its base model (about 17 GB) into `/models`.-`HF_TOKEN` is optional; arguments after the image name are appended to-`strands-decider serve`:+`HF_TOKEN` is optional. The image runs its own `serve.py` rather than+`strands-decider serve`: it merges the checkpoint's LoRA adapter into the base+weights once at load, which on an L4 gives the same answers in 0.26 s instead+of 0.42 s for a typical room and about 3.9 decisions a second instead of 2.4.+It's configured by env only (`STRANDS_CHECKPOINT`, `STRANDS_MODEL_NAME`,+`STRANDS_MAX_BATCH`):```docker build -t switch-decision-strands decision/model-servers/strands/diff --git a/decision/model-servers/strands/Dockerfile b/decision/model-servers/strands/Dockerfileindex 08fa0154e..f086600fb 100644--- a/decision/model-servers/strands/Dockerfile+++ b/decision/model-servers/strands/Dockerfile@@ -57,4 +57,5 @@ WORKDIR /home/decisionEXPOSE 8400HEALTHCHECK --interval=10s --timeout=5s --start-period=30m --retries=3 \CMD ["python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8400/health', timeout=3)"]-ENTRYPOINT ["sh", "-c", "exec strands-decider serve \"$STRANDS_CHECKPOINT\" --host 0.0.0.0 --port 8400 --device cuda --model-name \"$STRANDS_MODEL_NAME\" \"$@\"", "strands-decider"]+COPY --chown=10001:10001 serve.py /opt/strands-serve/serve.py+ENTRYPOINT ["python", "/opt/strands-serve/serve.py"]diff --git a/decision/model-servers/strands/serve.py b/decision/model-servers/strands/serve.pynew file mode 100644index 000000000..589e72a69--- /dev/null+++ b/decision/model-servers/strands/serve.py@@ -0,0 +1,89 @@+"""Serve a Strands Decider checkpoint over its SystemOne API, with the LoRA+adapter merged into the base weights.++ python serve.py++Reads STRANDS_CHECKPOINT, STRANDS_MODEL_NAME, STRANDS_PORT (default 8400),+STRANDS_HOST (default 0.0.0.0), STRANDS_DEVICE (default cuda) and+STRANDS_MAX_BATCH (default 32, requests evaluated together).++`strands-decider serve` keeps the adapter separate and applies it on every+forward pass. Merging it once at load gives the same answers (15/15, 36/40 and+56/60 on the evaluation sets, measured on an L4) in about 0.26 s instead of+0.42 s for a typical room, and about 3.9 decisions a second instead of 2.4.+The API is the same: POST /v1/systemone and GET /health.+"""++import os+import sys+import time++import torch+import uvicorn+from fastapi import FastAPI, HTTPException+from fastapi.responses import JSONResponse+from strands_decider.infer import EngineConfig, SystemOneEngine+from strands_decider.modeling import StrandsDeciderModel+from strands_decider.schema import SystemOneRequest++CHECKPOINT = os.environ["STRANDS_CHECKPOINT"]+MODEL_NAME = os.environ["STRANDS_MODEL_NAME"]+PORT = int(os.environ.get("STRANDS_PORT", "8400"))+HOST = os.environ.get("STRANDS_HOST", "0.0.0.0")+DEVICE = os.environ.get("STRANDS_DEVICE", "cuda")+MAX_BATCH = int(os.environ.get("STRANDS_MAX_BATCH", "32"))+++def load_engine() -> SystemOneEngine:+ started = time.monotonic()+ model = StrandsDeciderModel.load(CHECKPOINT)+ if not hasattr(model.torso, "merge_and_unload"):+ raise RuntimeError(+ f"{CHECKPOINT} has no LoRA adapter to merge; this server expects a "+ "Strands Decider checkpoint built on an adapter."+ )+ model.torso = model.torso.merge_and_unload()+ model.torso.to(DEVICE, dtype=torch.bfloat16)+ print(+ f"loaded {CHECKPOINT} as {MODEL_NAME} on {DEVICE} with the adapter merged, "+ f"in {time.monotonic() - started:.0f}s",+ flush=True,+ )+ return SystemOneEngine(+ model,+ EngineConfig(+ device=DEVICE,+ use_prefix_cache=True,+ model_name=MODEL_NAME,+ strict_window=False,+ max_batch=MAX_BATCH,+ max_batch_tokens=None,+ ),+ )+++engine = load_engine()+app = FastAPI(title="Strands Decider (adapter merged)")++[email protected]("/health")+def health() -> dict:+ return {"status": "ok", "model": MODEL_NAME, "checkpoint": CHECKPOINT, "adapter": "merged"}++[email protected]("/v1/systemone")+def systemone(request: SystemOneRequest) -> JSONResponse:+ try:+ response = engine.evaluate(request)+ except ValueError as exc:+ raise HTTPException(status_code=422, detail=str(exc)) from exc+ return JSONResponse(content=response.model_dump())+++if __name__ == "__main__":+ if len(sys.argv) > 1:+ raise SystemExit(+ "serve.py takes no arguments; configure it with STRANDS_* environment "+ f"variables (got {sys.argv[1:]})."+ )+ uvicorn.run(app, host=HOST, port=PORT, log_level="warning")diff --git a/deploy/remote/helm/switch/templates/decision/deployment.yaml b/deploy/remote/helm/switch/templates/decision/deployment.yamlindex 48de7090b..d7b795a71 100644--- a/deploy/remote/helm/switch/templates/decision/deployment.yaml+++ b/deploy/remote/helm/switch/templates/decision/deployment.yaml@@ -173,13 +173,6 @@ spec:- name: model-serverimage: {{ include "switch.image" (dict "global" .Values.global "image" $st.image) }}imagePullPolicy: {{ include "switch.imagePullPolicy" (dict "policy" $st.imagePullPolicy "image" $st.image) }}- {{- with $st.args }}- # Appended to the image's `strands-decider serve` command.- args:- {{- range . }}- - {{ . | quote }}- {{- end }}- {{- end }}env:- name: STRANDS_CHECKPOINTvalue: {{ required "decision.modelServer.strands.checkpoint is required" $st.checkpoint | quote }}diff --git a/deploy/remote/helm/switch/values.yaml b/deploy/remote/helm/switch/values.yamlindex 1706bed5a..31079c234 100644--- a/deploy/remote/helm/switch/values.yaml+++ b/deploy/remote/helm/switch/values.yaml@@ -939,11 +939,12 @@ decision:# first start with its base model (google/gemma-4-E4B-it, about 16 GB,# under the Gemma terms). Later starts only load it.checkpoint: StrandsAgents/strands-decider-E4B-gemma4-v1-2610- # --model-name, and DECISION_MODEL unless `model` above is set.+ # The name the server answers as, and DECISION_MODEL unless `model`+ # above is set.modelName: strands-decider-E4B- # Extra arguments for `strands-decider serve` (e.g. --max-batch 8).- args: []- # Extra container env, as Kubernetes env entries.+ # Extra container env, as Kubernetes env entries. The server (the image's+ # serve.py, which merges the adapter into the base weights) is+ # configured only by env, e.g. STRANDS_MAX_BATCH (default 32).env: []# The port opens only once the model is loaded: the first start# downloads about 16 GB and then loads it (a few minutes with 32 GiB of