Skip to content

Self-improving AI Employees for everyone

fix(sarvam): ship the Starter-tier max_tokens cap so requests succeed

trohitg703fee42b40b
diff --git a/docs-internal/node-logic-flows/ai_chat_models/sarvamChatModel.md b/docs-internal/node-logic-flows/ai_chat_models/sarvamChatModel.md
index 3dad668a..8d59cac8 100644
--- a/docs-internal/node-logic-flows/ai_chat_models/sarvamChatModel.md
+++ b/docs-internal/node-logic-flows/ai_chat_models/sarvamChatModel.md
@@ -31,7 +31,7 @@ subclass); reasoning is on by default server-side.
| `system_prompt` | string | `""` | no | - | System prompt |
| `model` | string | `""` (injected) | no | - | `sarvam-105b` / `sarvam-30b` |
| `temperature` | number\|null | `null` | no | - | 0-2. Sarvam's own default is 0.5 with reasoning on, 0.2 without |
-| `max_tokens` | number\|null | `null` | no | - | Defaults to the `llm_defaults.json` value (65536 / 32768) |
+| `max_tokens` | number\|null | `null` | no | - | Defaults to the `llm_defaults.json` value (4096 — the Starter-tier cap). Raising it here does nothing unless the JSON is raised too |
| `top_p` | number\|null | `1.0` | no | - | |
| `frequency_penalty` | number\|null | `0.0` | no | - | -2.0 to 2.0 |
| `presence_penalty` | number\|null | `0.0` | no | - | -2.0 to 2.0 |
@@ -107,7 +107,7 @@ flowchart TD
## Edge cases & known limits
- **No `/v1/models` endpoint.** Verified against <https://docs.sarvam.ai/openapi.json> (25 paths, none for model listing). Unhandled, the 404 surfaces as an `openai.OpenAIError` -> `NodeUserError`, which `AIService.fetch_models` re-raises *before* its curated fallback — breaking credential validation and the model dropdown for a valid key. The `supports_model_listing: false` flag routes around this generically.
-- **`max_output_tokens` is half the context window** (65536 / 32768), not Sarvam's documented Business-tier ceilings (128000 / 64000). Those are ~98% of the context window and `resolve_max_tokens` uses the value as both the unset default and the clamp ceiling, so a request at the documented ceiling could not fit a prompt.
+- **The output cap is a subscription tier, not a model limit.** Sarvam allows 4096 output tokens on Starter, 16384 / 8192 on Pro and 128000 / 64000 on Business, and 400s on anything above your tier ("max_tokens (65536) exceeds the maximum allowed for sarvam-105b for your subscription tier (starter): 4096"). `resolve_max_tokens` uses the `llm_defaults.json` number as both the unset default and the clamp ceiling, so it ships as the Starter cap (4096) — the only value that works on every account. Pro/Business users must raise it in the JSON; raising `max_tokens` on the node alone is clamped straight back down.
- **`popular_models` is `[]`** per the >=1M-context policy (Sarvam maxes at 131072); the dropdown falls through to the `max_output_tokens` keys, so both models still appear.
- **`sarvam-m` is deprecated** and removed from the API — deliberately absent from the config.
- **`wiki_grounding` is not exposed.** Sarvam accepts it, but an extra `Params` field cannot reach the API today: `execute_chat` reads a closed key set off `flattened`. See [Sarvam AI Service](../../sarvam_service.md#known-limits).
diff --git a/docs-internal/sarvam_service.md b/docs-internal/sarvam_service.md
index 7dfa0349..7f08f2ce 100644
--- a/docs-internal/sarvam_service.md
+++ b/docs-internal/sarvam_service.md
@@ -31,18 +31,49 @@ registered declaratively in
| Model | Context | `max_output_tokens` in JSON | Notes |
|---|---|---|---|
-| `sarvam-105b` | 131072 | 65536 | Flagship, `default_model` |
-| `sarvam-30b` | 65536 | 32768 | Cheaper tier |
+| `sarvam-105b` | 131072 | 4096 | Flagship, `default_model` |
+| `sarvam-30b` | 65536 | 4096 | Cheaper tier |
`sarvam-m` is deprecated and removed from the API; it is deliberately absent.
-**`max_output_tokens` is half the context window, not Sarvam's documented
-ceiling.** Sarvam publishes per-tier output caps (Business: 128000 for the
-105b, 64000 for the 30b), but those are ~98% of the context window, and
-`resolve_max_tokens` uses this value as *both* the default budget when a node
-leaves `max_tokens` unset *and* the clamp ceiling — a request at the documented
-ceiling could not fit any prompt. Half the window keeps paid tiers unclamped
-and usable. The tier table is recorded in `_max_output_note`.
+### The output cap is a billing tier, not a model limit
+
+This is the one provider here whose output ceiling depends on the **account**
+rather than the model. Sarvam publishes:
+
+| Tier | `sarvam-105b` | `sarvam-30b` |
+|---|---|---|
+| Starter | 4096 | 4096 |
+| Pro | 16384 | 8192 |
+| Business | 128000 | 64000 |
+
+Exceeding your tier is a hard 400:
+
+```
+max_tokens (65536) exceeds the maximum allowed for sarvam-105b
+for your subscription tier (starter): 4096.
+```
+
+`resolve_max_tokens` ([`llm/config.py`](../server/services/llm/config.py))
+uses this number as **both** the default budget when a node leaves
+`max_tokens` unset **and** the ceiling it clamps larger user values down to.
+One static value therefore has to hold for every account, so the shipped
+figure is the **Starter cap** — the only one that succeeds on all tiers.
+Sarvam's own API default is 2048, so 4096 is already generous.
+
+**On Pro or Business**, raise these numbers in `llm_defaults.json` to your
+tier's ceiling. Setting `max_tokens` on the node is *not* enough — the
+resolver clamps it straight back down to whatever is written here. Do not
+simply use the Business figures as the default: they are ~98% of each context
+window and leave no room for a prompt.
+
+The context windows are untouched by any of this — those are real model
+properties (131072 / 65536) and drive compaction thresholds.
+
+Pinned by `TestSarvamSubscriptionTierCap` in
+[`tests/llm/test_max_tokens_resolution.py`](../server/tests/llm/test_max_tokens_resolution.py),
+which also guards against the assertions passing off the registry's hardcoded
+fallbacks.
### Reasoning is entirely JSON-driven
diff --git a/server/config/llm_defaults.json b/server/config/llm_defaults.json
index 6583b904..70024dc5 100644
--- a/server/config/llm_defaults.json
+++ b/server/config/llm_defaults.json
@@ -326,11 +326,11 @@
"max_tokens_param": "max_tokens",
"supported_params": ["temperature", "max_tokens", "top_p", "frequency_penalty", "presence_penalty", "stop", "seed", "n", "tools", "tool_choice", "response_format", "reasoning_effort"],
"max_output_tokens": {
- "sarvam-105b": 65536,
- "sarvam-30b": 32768,
- "_default": 8192
+ "sarvam-105b": 4096,
+ "sarvam-30b": 4096,
+ "_default": 4096
},
- "_max_output_note": "Sarvam documents per-tier output ceilings of Starter 4096 / Pro 16384 / Business 128000 (105b) and 4096 / 8192 / 64000 (30b). The Business figures are ~98% of each context window, and resolve_max_tokens uses these values BOTH as the default budget when the node leaves max_tokens unset AND as the clamp ceiling — a request at that budget could not fit any prompt. Half the window keeps paid tiers unclamped while staying usable; same shape as groq's gpt-oss-120b (131072 ctx / 65536 out).",
+ "_max_output_note": "SUBSCRIPTION-TIER CAP, not a model limit — this is the one provider here whose output ceiling depends on the ACCOUNT rather than the model. Sarvam documents Starter 4096 / Pro 16384 / Business 128000 for sarvam-105b and 4096 / 8192 / 64000 for sarvam-30b, and rejects anything above your tier with a 400 ('max_tokens (N) exceeds the maximum allowed ... for your subscription tier'). services/llm/config.py::resolve_max_tokens uses this number BOTH as the default budget when a node leaves max_tokens unset AND as the ceiling it clamps larger user values down to, so a single value has to hold for every account — hence the Starter cap, the only figure that succeeds on all tiers. Sarvam's own API default is 2048, so 4096 is already generous. ON PRO OR BUSINESS: raise these to your tier's ceiling (setting max_tokens on the node alone is NOT enough — resolve_max_tokens clamps it back down to whatever is written here). Do not use the Business figures as a default: they are ~98% of each context window, leaving no room for a prompt.",
"context_length": {
"sarvam-105b": 131072,
"sarvam-30b": 65536,
diff --git a/server/tests/llm/test_max_tokens_resolution.py b/server/tests/llm/test_max_tokens_resolution.py
index aa4dd835..c645ccfe 100644
--- a/server/tests/llm/test_max_tokens_resolution.py
+++ b/server/tests/llm/test_max_tokens_resolution.py
@@ -129,6 +129,82 @@ class TestRegistryAliasNormalization:
assert info.max_output_tokens == 65536
+class TestSarvamSubscriptionTierCap:
+ """Sarvam's output ceiling is set by the ACCOUNT tier, not the model.
+
+ Every other provider's max_output_tokens is a model capability, so a
+ single JSON number is correct for all users. Sarvam rejects anything
+ above the caller's tier with a 400 ('max_tokens (65536) exceeds the
+ maximum allowed for sarvam-105b for your subscription tier
+ (starter): 4096'), and since resolve_max_tokens uses this value both
+ as the unset default and as the clamp ceiling, the shipped number has
+ to be the Starter cap -- the only one that succeeds on every tier.
+
+ A well-meaning "these models support way more" edit would silently
+ break every Starter and Pro account, so it is pinned here.
+ """
+
+ STARTER_CAP = 4096
+
+ @pytest.fixture
+ def loaded_registry(self):
+ """A registry with llm_defaults.json actually loaded.
+
+ ``get_model_registry()`` only reads the JSON in ``startup()``, and
+ an unloaded registry falls through to hardcoded 4096 / 128000
+ defaults -- which would make every assertion below pass no matter
+ what the config says. Patching in a locally loaded instance keeps
+ the test honest without mutating the process-wide singleton.
+ """
+ from services.model_registry import ModelRegistryService
+
+ svc = ModelRegistryService()
+ svc._load_llm_defaults()
+ with patch("services.model_registry.get_model_registry", return_value=svc):
+ yield svc
+
+ def test_fixture_actually_loaded_the_config(self, loaded_registry):
+ """Guard against the other tests passing off the hard fallbacks.
+
+ 128000 is the hardcoded context fallback; reading 131072 proves
+ the sarvam block was parsed.
+ """
+ assert loaded_registry.get_context_length("sarvam-105b", "sarvam") == 131072
+
+ @pytest.mark.parametrize("model", ["sarvam-105b", "sarvam-30b"])
+ def test_shipped_ceiling_is_the_starter_tier_cap(self, model):
+ from services.llm.config import LLM_DEFAULTS
+
+ caps = LLM_DEFAULTS["providers"]["sarvam"]["max_output_tokens"]
+ assert caps[model] <= self.STARTER_CAP, (
+ f"{model} ships max_output_tokens={caps[model]}, above the Starter "
+ f"tier cap of {self.STARTER_CAP}. Sarvam 400s on that for any "
+ "account below the matching tier, and resolve_max_tokens treats "
+ "this value as a hard ceiling."
+ )
+ assert caps["_default"] <= self.STARTER_CAP
+
+ @pytest.mark.parametrize("model", ["sarvam-105b", "sarvam-30b"])
+ def test_unset_max_tokens_resolves_within_the_cap(self, model, loaded_registry):
+ """The default budget must be sendable as-is on a Starter account."""
+ from services.llm.config import resolve_max_tokens
+
+ assert resolve_max_tokens({}, model, "sarvam") <= self.STARTER_CAP
+
+ def test_oversized_request_is_clamped_not_forwarded(self, loaded_registry):
+ from services.llm.config import resolve_max_tokens
+
+ assert resolve_max_tokens(
+ {"max_tokens": 65536}, "sarvam-105b", "sarvam"
+ ) <= self.STARTER_CAP
+
+ def test_context_length_is_untouched_by_the_tier_cap(self, loaded_registry):
+ """The tier limits OUTPUT only -- the context window is a real
+ model property and must not be shrunk along with it."""
+ assert loaded_registry.get_context_length("sarvam-105b", "sarvam") == 131072
+ assert loaded_registry.get_context_length("sarvam-30b", "sarvam") == 65536
+
+
class TestStandaloneWorkerRegistryStartup:
def test_standalone_worker_loads_model_registry(self):
# Without startup(), a standalone worker's registry is empty and