Self-improving AI Employees for everyone
Python ai-agentsai-employeeai-employeesassistantautonomous-agentsco-employeescompanygrok-botinstinctloop-agentsmuseown-your-datapersonalself-hosted
fix(sarvam): ship the Starter-tier max_tokens cap so requests succeed
trohitg703fee42b40b
diff --git a/docs-internal/node-logic-flows/ai_chat_models/sarvamChatModel.md b/docs-internal/node-logic-flows/ai_chat_models/sarvamChatModel.mdindex 3dad668a..8d59cac8 100644--- a/docs-internal/node-logic-flows/ai_chat_models/sarvamChatModel.md+++ b/docs-internal/node-logic-flows/ai_chat_models/sarvamChatModel.md@@ -31,7 +31,7 @@ subclass); reasoning is on by default server-side.| `system_prompt` | string | `""` | no | - | System prompt || `model` | string | `""` (injected) | no | - | `sarvam-105b` / `sarvam-30b` || `temperature` | number\|null | `null` | no | - | 0-2. Sarvam's own default is 0.5 with reasoning on, 0.2 without |-| `max_tokens` | number\|null | `null` | no | - | Defaults to the `llm_defaults.json` value (65536 / 32768) |+| `max_tokens` | number\|null | `null` | no | - | Defaults to the `llm_defaults.json` value (4096 — the Starter-tier cap). Raising it here does nothing unless the JSON is raised too || `top_p` | number\|null | `1.0` | no | - | || `frequency_penalty` | number\|null | `0.0` | no | - | -2.0 to 2.0 || `presence_penalty` | number\|null | `0.0` | no | - | -2.0 to 2.0 |@@ -107,7 +107,7 @@ flowchart TD## Edge cases & known limits- **No `/v1/models` endpoint.** Verified against <https://docs.sarvam.ai/openapi.json> (25 paths, none for model listing). Unhandled, the 404 surfaces as an `openai.OpenAIError` -> `NodeUserError`, which `AIService.fetch_models` re-raises *before* its curated fallback — breaking credential validation and the model dropdown for a valid key. The `supports_model_listing: false` flag routes around this generically.-- **`max_output_tokens` is half the context window** (65536 / 32768), not Sarvam's documented Business-tier ceilings (128000 / 64000). Those are ~98% of the context window and `resolve_max_tokens` uses the value as both the unset default and the clamp ceiling, so a request at the documented ceiling could not fit a prompt.+- **The output cap is a subscription tier, not a model limit.** Sarvam allows 4096 output tokens on Starter, 16384 / 8192 on Pro and 128000 / 64000 on Business, and 400s on anything above your tier ("max_tokens (65536) exceeds the maximum allowed for sarvam-105b for your subscription tier (starter): 4096"). `resolve_max_tokens` uses the `llm_defaults.json` number as both the unset default and the clamp ceiling, so it ships as the Starter cap (4096) — the only value that works on every account. Pro/Business users must raise it in the JSON; raising `max_tokens` on the node alone is clamped straight back down.- **`popular_models` is `[]`** per the >=1M-context policy (Sarvam maxes at 131072); the dropdown falls through to the `max_output_tokens` keys, so both models still appear.- **`sarvam-m` is deprecated** and removed from the API — deliberately absent from the config.- **`wiki_grounding` is not exposed.** Sarvam accepts it, but an extra `Params` field cannot reach the API today: `execute_chat` reads a closed key set off `flattened`. See [Sarvam AI Service](../../sarvam_service.md#known-limits).diff --git a/docs-internal/sarvam_service.md b/docs-internal/sarvam_service.mdindex 7dfa0349..7f08f2ce 100644--- a/docs-internal/sarvam_service.md+++ b/docs-internal/sarvam_service.md@@ -31,18 +31,49 @@ registered declaratively in| Model | Context | `max_output_tokens` in JSON | Notes ||---|---|---|---|-| `sarvam-105b` | 131072 | 65536 | Flagship, `default_model` |-| `sarvam-30b` | 65536 | 32768 | Cheaper tier |+| `sarvam-105b` | 131072 | 4096 | Flagship, `default_model` |+| `sarvam-30b` | 65536 | 4096 | Cheaper tier |`sarvam-m` is deprecated and removed from the API; it is deliberately absent.-**`max_output_tokens` is half the context window, not Sarvam's documented-ceiling.** Sarvam publishes per-tier output caps (Business: 128000 for the-105b, 64000 for the 30b), but those are ~98% of the context window, and-`resolve_max_tokens` uses this value as *both* the default budget when a node-leaves `max_tokens` unset *and* the clamp ceiling — a request at the documented-ceiling could not fit any prompt. Half the window keeps paid tiers unclamped-and usable. The tier table is recorded in `_max_output_note`.+### The output cap is a billing tier, not a model limit++This is the one provider here whose output ceiling depends on the **account**+rather than the model. Sarvam publishes:++| Tier | `sarvam-105b` | `sarvam-30b` |+|---|---|---|+| Starter | 4096 | 4096 |+| Pro | 16384 | 8192 |+| Business | 128000 | 64000 |++Exceeding your tier is a hard 400:++```+max_tokens (65536) exceeds the maximum allowed for sarvam-105b+for your subscription tier (starter): 4096.+```++`resolve_max_tokens` ([`llm/config.py`](../server/services/llm/config.py))+uses this number as **both** the default budget when a node leaves+`max_tokens` unset **and** the ceiling it clamps larger user values down to.+One static value therefore has to hold for every account, so the shipped+figure is the **Starter cap** — the only one that succeeds on all tiers.+Sarvam's own API default is 2048, so 4096 is already generous.++**On Pro or Business**, raise these numbers in `llm_defaults.json` to your+tier's ceiling. Setting `max_tokens` on the node is *not* enough — the+resolver clamps it straight back down to whatever is written here. Do not+simply use the Business figures as the default: they are ~98% of each context+window and leave no room for a prompt.++The context windows are untouched by any of this — those are real model+properties (131072 / 65536) and drive compaction thresholds.++Pinned by `TestSarvamSubscriptionTierCap` in+[`tests/llm/test_max_tokens_resolution.py`](../server/tests/llm/test_max_tokens_resolution.py),+which also guards against the assertions passing off the registry's hardcoded+fallbacks.### Reasoning is entirely JSON-drivendiff --git a/server/config/llm_defaults.json b/server/config/llm_defaults.jsonindex 6583b904..70024dc5 100644--- a/server/config/llm_defaults.json+++ b/server/config/llm_defaults.json@@ -326,11 +326,11 @@"max_tokens_param": "max_tokens","supported_params": ["temperature", "max_tokens", "top_p", "frequency_penalty", "presence_penalty", "stop", "seed", "n", "tools", "tool_choice", "response_format", "reasoning_effort"],"max_output_tokens": {- "sarvam-105b": 65536,- "sarvam-30b": 32768,- "_default": 8192+ "sarvam-105b": 4096,+ "sarvam-30b": 4096,+ "_default": 4096},- "_max_output_note": "Sarvam documents per-tier output ceilings of Starter 4096 / Pro 16384 / Business 128000 (105b) and 4096 / 8192 / 64000 (30b). The Business figures are ~98% of each context window, and resolve_max_tokens uses these values BOTH as the default budget when the node leaves max_tokens unset AND as the clamp ceiling — a request at that budget could not fit any prompt. Half the window keeps paid tiers unclamped while staying usable; same shape as groq's gpt-oss-120b (131072 ctx / 65536 out).",+ "_max_output_note": "SUBSCRIPTION-TIER CAP, not a model limit — this is the one provider here whose output ceiling depends on the ACCOUNT rather than the model. Sarvam documents Starter 4096 / Pro 16384 / Business 128000 for sarvam-105b and 4096 / 8192 / 64000 for sarvam-30b, and rejects anything above your tier with a 400 ('max_tokens (N) exceeds the maximum allowed ... for your subscription tier'). services/llm/config.py::resolve_max_tokens uses this number BOTH as the default budget when a node leaves max_tokens unset AND as the ceiling it clamps larger user values down to, so a single value has to hold for every account — hence the Starter cap, the only figure that succeeds on all tiers. Sarvam's own API default is 2048, so 4096 is already generous. ON PRO OR BUSINESS: raise these to your tier's ceiling (setting max_tokens on the node alone is NOT enough — resolve_max_tokens clamps it back down to whatever is written here). Do not use the Business figures as a default: they are ~98% of each context window, leaving no room for a prompt.","context_length": {"sarvam-105b": 131072,"sarvam-30b": 65536,diff --git a/server/tests/llm/test_max_tokens_resolution.py b/server/tests/llm/test_max_tokens_resolution.pyindex aa4dd835..c645ccfe 100644--- a/server/tests/llm/test_max_tokens_resolution.py+++ b/server/tests/llm/test_max_tokens_resolution.py@@ -129,6 +129,82 @@ class TestRegistryAliasNormalization:assert info.max_output_tokens == 65536+class TestSarvamSubscriptionTierCap:+ """Sarvam's output ceiling is set by the ACCOUNT tier, not the model.++ Every other provider's max_output_tokens is a model capability, so a+ single JSON number is correct for all users. Sarvam rejects anything+ above the caller's tier with a 400 ('max_tokens (65536) exceeds the+ maximum allowed for sarvam-105b for your subscription tier+ (starter): 4096'), and since resolve_max_tokens uses this value both+ as the unset default and as the clamp ceiling, the shipped number has+ to be the Starter cap -- the only one that succeeds on every tier.++ A well-meaning "these models support way more" edit would silently+ break every Starter and Pro account, so it is pinned here.+ """++ STARTER_CAP = 4096++ @pytest.fixture+ def loaded_registry(self):+ """A registry with llm_defaults.json actually loaded.++ ``get_model_registry()`` only reads the JSON in ``startup()``, and+ an unloaded registry falls through to hardcoded 4096 / 128000+ defaults -- which would make every assertion below pass no matter+ what the config says. Patching in a locally loaded instance keeps+ the test honest without mutating the process-wide singleton.+ """+ from services.model_registry import ModelRegistryService++ svc = ModelRegistryService()+ svc._load_llm_defaults()+ with patch("services.model_registry.get_model_registry", return_value=svc):+ yield svc++ def test_fixture_actually_loaded_the_config(self, loaded_registry):+ """Guard against the other tests passing off the hard fallbacks.++ 128000 is the hardcoded context fallback; reading 131072 proves+ the sarvam block was parsed.+ """+ assert loaded_registry.get_context_length("sarvam-105b", "sarvam") == 131072++ @pytest.mark.parametrize("model", ["sarvam-105b", "sarvam-30b"])+ def test_shipped_ceiling_is_the_starter_tier_cap(self, model):+ from services.llm.config import LLM_DEFAULTS++ caps = LLM_DEFAULTS["providers"]["sarvam"]["max_output_tokens"]+ assert caps[model] <= self.STARTER_CAP, (+ f"{model} ships max_output_tokens={caps[model]}, above the Starter "+ f"tier cap of {self.STARTER_CAP}. Sarvam 400s on that for any "+ "account below the matching tier, and resolve_max_tokens treats "+ "this value as a hard ceiling."+ )+ assert caps["_default"] <= self.STARTER_CAP++ @pytest.mark.parametrize("model", ["sarvam-105b", "sarvam-30b"])+ def test_unset_max_tokens_resolves_within_the_cap(self, model, loaded_registry):+ """The default budget must be sendable as-is on a Starter account."""+ from services.llm.config import resolve_max_tokens++ assert resolve_max_tokens({}, model, "sarvam") <= self.STARTER_CAP++ def test_oversized_request_is_clamped_not_forwarded(self, loaded_registry):+ from services.llm.config import resolve_max_tokens++ assert resolve_max_tokens(+ {"max_tokens": 65536}, "sarvam-105b", "sarvam"+ ) <= self.STARTER_CAP++ def test_context_length_is_untouched_by_the_tier_cap(self, loaded_registry):+ """The tier limits OUTPUT only -- the context window is a real+ model property and must not be shrunk along with it."""+ assert loaded_registry.get_context_length("sarvam-105b", "sarvam") == 131072+ assert loaded_registry.get_context_length("sarvam-30b", "sarvam") == 65536++class TestStandaloneWorkerRegistryStartup:def test_standalone_worker_loads_model_registry(self):# Without startup(), a standalone worker's registry is empty and