Skip to content

Compress tool outputs, logs, files, and RAG chunks before they reach the LLM. 20% fewer tokens for coding agents, 60-95% fewer tokens for JSON, same answers. Library, proxy, MCP server.

docs(configuration): document LiteLLM as the source for limits and prices (#2781)

Tejas Chopra95ae69a85f5a
diff --git a/docs/content/docs/configuration.mdx b/docs/content/docs/configuration.mdx
index 70f013efc..b77aedd0e 100644
--- a/docs/content/docs/configuration.mdx
+++ b/docs/content/docs/configuration.mdx
@@ -549,8 +549,11 @@ Configure context limits and pricing for new or custom models:
},
"openai": {
"context_limits": {
- "gpt-5": 256000,
- "ft:gpt-4o:my-org": 128000
+ "ft:gpt-4o:my-org": 128000,
+ "my-private-deployment": 200000
+ },
+ "pricing": {
+ "my-private-deployment": [5.00, 15.00]
}
}
}
@@ -561,18 +564,61 @@ Save as `${HEADROOM_CONFIG_DIR}/models.json` (defaults to
JSON string or file path. Installs that still have
`~/.headroom/models.json` (the legacy location) continue to work.
-Settings are resolved in this order (later overrides earlier):
+<Callout type="info">
+ Configure **models Headroom can't already look up** — fine-tunes, private
+ deployments, gateway aliases. Public models are resolved from LiteLLM (see
+ below), so pinning one here means maintaining a number yourself that would
+ otherwise stay current on its own.
+</Callout>
+
+### Where limits and prices come from
+
+Headroom consults the **installed LiteLLM model database** (`litellm.model_cost`)
+for public model metadata. That data ships with the LiteLLM package rather than
+being fetched at runtime, so it is as current as your installed LiteLLM version
+and refreshes when you upgrade it. Nothing here makes a network call.
+
+Limits and prices do **not** use the same precedence. Both are first-match-wins:
+
+**Context limits**
+
+1. **Explicit configuration and the built-in table**, checked together — they are
+ merged into one mapping, so an exact match in either returns immediately, then
+ partial/prefix matches.
+2. **LiteLLM** (`max_input_tokens`) — reached only for models the previous step
+ didn't match.
+3. **Pattern inference**, then a generic default.
-1. Built-in defaults
-2. `${HEADROOM_CONFIG_DIR}/models.json` (new canonical location); falls
- back to `~/.headroom/models.json` (legacy) when the canonical file
- is absent
-3. `HEADROOM_MODEL_LIMITS` environment variable
-4. SDK constructor arguments
+**Pricing**
+
+1. **Explicit configuration** — a value you configured is a decision, so it beats
+ everything below.
+2. **LiteLLM.**
+3. **Built-in table**, then pattern inference, then a generic default.
+
+<Callout type="warn">
+ The asymmetry is deliberate, not an oversight. For limits, a built-in entry
+ outranks LiteLLM because LiteLLM reports a model's maximum *capability* while
+ Headroom needs its *effective default*. `claude-sonnet-4-20250514` is the
+ concrete case: LiteLLM reports 1,000,000, but that window is an opt-in beta, so
+ the built-in 200,000 is the safe assumption for a client that has not enabled
+ it. Pricing has no equivalent capability-vs-default split, so there LiteLLM
+ wins outright.
+</Callout>
+
+Limits are an **input** budget (`max_input_tokens`), not the total window: `gpt-5`
+resolves to 272K input, not the 400K total (272K in + 128K out).
+
+LiteLLM also resolves gateway-routed names — `azure/...`, `bedrock/...`,
+`vertex_ai/...`, `groq/...` — which the built-in tables never covered.
+
+The built-in tables additionally cover installs where LiteLLM is unavailable: the
+dependency is gated `python_version < '3.14'`.
### Pattern-Based Inference
-Unknown models are automatically inferred from naming patterns:
+Models that reach step 3 and match no table entry are inferred from naming
+patterns:
| Pattern | Inferred Settings |
|---------|-------------------|
@@ -580,7 +626,12 @@ Unknown models are automatically inferred from naming patterns:
| `*sonnet*` | 200K context, Sonnet-tier pricing |
| `*haiku*` | 200K context, Haiku-tier pricing |
| `gpt-4o*` | 128K context, GPT-4o pricing |
-| `o1*`, `o3*` | 200K context, reasoning model pricing |
+| `gpt-4.1*` | ~1M context, GPT-4.1 pricing |
+| `gpt-5*` | 272K input, GPT-5 pricing |
+| `o1*`, `o3*`, `o4*` | 200K context, reasoning model pricing |
+
+Matching prefers the **most specific** prefix, so `gpt-4.1` is not treated as
+`gpt-4` (which would give it an 8K limit and legacy `gpt-4` pricing).
## Provider-Specific Settings
diff --git a/wiki/configuration.md b/wiki/configuration.md
index 35684298a..11ce3894c 100644
--- a/wiki/configuration.md
+++ b/wiki/configuration.md
@@ -302,13 +302,55 @@ Configure context limits and pricing for new or custom models. Useful when:
### Configuration Methods
-Settings are resolved in this order (later overrides earlier):
-1. Built-in defaults
-2. `${HEADROOM_CONFIG_DIR}/models.json` (defaults to
+Headroom reads public model metadata from the **installed LiteLLM model database**
+(`litellm.model_cost`). That data ships with the LiteLLM package rather than being
+fetched at runtime, so it is as current as your installed LiteLLM version and
+refreshes when you upgrade it. No network call is made.
+
+Explicit configuration comes from three places, later overriding earlier:
+
+1. `${HEADROOM_CONFIG_DIR}/models.json` (defaults to
`~/.headroom/config/models.json`); falls back to the legacy location
`~/.headroom/models.json` when the canonical file is absent
-3. `HEADROOM_MODEL_LIMITS` environment variable
-4. SDK constructor arguments
+2. `HEADROOM_MODEL_LIMITS` environment variable
+3. SDK constructor arguments
+
+Limits and prices then resolve with **different** precedence. Both are
+first-match-wins:
+
+**Context limits**
+
+1. **Explicit configuration and the built-in table, checked together** — they are
+ merged into one mapping, so an exact match in either returns immediately,
+ followed by partial/prefix matches.
+2. **LiteLLM** (`max_input_tokens`) — reached only for models step 1 didn't match.
+3. **Pattern inference**, then a generic default.
+
+**Pricing**
+
+1. **Explicit configuration** — a configured value is a decision, so it beats
+ everything below.
+2. **LiteLLM.**
+3. **Built-in table**, then pattern inference, then a generic default.
+
+The asymmetry is deliberate. For limits a built-in entry outranks LiteLLM because
+LiteLLM reports a model's maximum *capability* while Headroom needs its *effective
+default*: LiteLLM gives `claude-sonnet-4-20250514` 1,000,000, but that window is
+an opt-in beta, so the built-in 200,000 is the safe assumption for a client that
+has not enabled it. Pricing has no capability-vs-default split, so there LiteLLM
+wins outright.
+
+Limits are an **input** budget (`max_input_tokens`), not the total window: `gpt-5`
+resolves to 272K input, not the 400K total (272K in + 128K out).
+
+LiteLLM also resolves gateway-routed names (`azure/...`, `bedrock/...`,
+`vertex_ai/...`, `groq/...`) that the built-in tables never covered, and the
+built-in tables additionally cover installs where LiteLLM is unavailable (the
+dependency is gated `python_version < '3.14'`).
+
+Configure only models Headroom can't already look up — fine-tunes, private
+deployments, gateway aliases. Pinning a public model's *price* here means
+maintaining a number yourself that would otherwise stay current.
### Config File Format
@@ -331,11 +373,11 @@ Create `~/.headroom/models.json`:
},
"openai": {
"context_limits": {
- "gpt-5": 256000,
- "ft:gpt-4o:my-org": 128000
+ "ft:gpt-4o:my-org": 128000,
+ "my-private-deployment": 200000
},
"pricing": {
- "gpt-5": [5.00, 15.00]
+ "my-private-deployment": [5.00, 15.00]
}
}
}